We introduce SODA, a generalization of Optimistic Dual Averaging, which provides a common perspective on state-of-the-art optimizers like Muon, Lion, AdEMAMix and NAdam, showing that they can all be viewed as optimistic instances of this framework. Based on this framing, we propose a practical SODA wrapper for any base optimizer that eliminates weight decay tuning through a theoretically-grounded 1/k decay schedule. Empirical results across various scales and training horizons show that SODA consistently improves performance without any additional hyperparameter tuning.
Cite
@article{arxiv.2605.11172,
title = {Optimistic Dual Averaging Unifies Modern Optimizers},
author = {Thomas Pethick and Wanyun Xie and Roman Machacek and Volkan Cevher},
journal= {arXiv preprint arXiv:2605.11172},
year = {2026}
}