Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE),a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures.
@article{arxiv.2603.04971,
title = {Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation},
author = {Yilong Chen and Naibin Gu and Junyuan Shang and Zhenyu Zhang and Yuchen Feng and Jiawei Sheng and Tingwen Liu and Shuohuan Wang and Yu Sun and Hua Wu and Haifeng Wang},
journal= {arXiv preprint arXiv:2603.04971},
year = {2026}
}