Efficient Distributed Orthonormal Optimizers for Large-Scale Training
614
8.9
Next
46 days – 43052:29Popular
186 days – 2 5551:56:28
270 days – 7151:16:09Published on 6 Mar 2026, 15:16
Speaker: Kwangjun Ahn, Microsoft Research
I delivered a 50-minute technical talk on recent advances in orthonormal update methods for large-scale AI model training. This topic has been rapidly gaining attention in the community, emerging as a strong successor to AdamW following the success of orthonormal optimizers in training production-scale models such as Kimi-K2 and GLM-4.5.
The talk centered on the design and practice of orthonormal updates, with a focus on optimizers such as Muon and Dion2. While I briefly discussed their theoretical foundations, the emphasis was on practical usage: how to integrate these optimizers into modern training pipelines, interpret their algorithmic components, and leverage the implementation guidelines provided in our open-source codebase at github.com/microsoft/dion
I delivered a 50-minute technical talk on recent advances in orthonormal update methods for large-scale AI model training. This topic has been rapidly gaining attention in the community, emerging as a strong successor to AdamW following the success of orthonormal optimizers in training production-scale models such as Kimi-K2 and GLM-4.5.
The talk centered on the design and practice of orthonormal updates, with a focus on optimizers such as Muon and Dion2. While I briefly discussed their theoretical foundations, the emphasis was on practical usage: how to integrate these optimizers into modern training pipelines, interpret their algorithmic components, and leverage the implementation guidelines provided in our open-source codebase at github.com/microsoft/dion
Fresh videos
1 day – 156 71713:01
3 days – 2 742 5420:16
4 days – 15 3530:56
4 days – 2 203 2420:57
6 days – 1 05930:04
6 days – 24 4780:30Random videos
42 days – 1 8650:48
338 days – 356 20112:45
18.10.24 – 4 94034:51
20.08.22 – 410 73030:52
1 day – 9 06524:48
3 days – 24 1220:23
6 days – 46 9395:37
8 days – 1 7840:10
11 days – 4 8511:17
14 days – 17954:36
14 days – 11 9581:42
