Orthonormalized updates accelerate training, improve stability, and enable robust hyperparameter transfer, but existing methods like Muon rely on dense matrix operations that clash with sharded weights in large-scale LLM training, causing high compute and communication cost. We introduce Dion (Distributed Orthonormalization), a scalable and efficient update rule that replaces Newton-Schulz iteration with amortized power iteration on a momentum buffer, avoiding full-matrix reconstruction and integrating cleanly with weight sharding. The rank-fraction parameter with error feedback enables low-rank updates that balance quality with significant cost savings. On language models from 160M to 3B parameters, Dion retains the benefits of orthonormalized updates, while markedly reducing wall-clock time at scale, making it a practical optimizer for next-generation foundation models. Code is available at: https://github.com/microsoft/dion/
@article{arxiv.2504.05295,
title = {Dion: Distributed Orthonormalized Updates},
author = {Kwangjun Ahn and Byron Xu and Natalie Abreu and Ying Fan and Gagik Magakyan and Pratyusha Sharma and Zheng Zhan and John Langford},
journal= {arXiv preprint arXiv:2504.05295},
year = {2025}
}