The Mixture-of-Experts (MoE) architecture is a powerful technique for scaling language models, yet it often suffers from expert homogenization, where experts learn redundant functionalities, thereby limiting MoE's full potential. To address this, we introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts. Our method incorporates a label-driven auxiliary loss that leverages domain labels inherent in pre-training corpora to maximize the Jensen-Shannon Divergence between the expert routing distributions of different data domains. This optimization objective guides the model to develop diverged routing policies for varied domains and closer routing policies for the same domain, which leads to emergent and organized expert specialization. We validate our approach by pre-training MoE models of up to 15 billion parameters from scratch. Experimental results demonstrate that models trained with Expert Divergence Learning not only achieve a lower language modeling loss but also exhibit significant performance improvements across a diverse range of downstream benchmarks. Further analysis confirms that our method effectively mitigates expert homogenization and brings greater functional specialization, all with negligible computational overhead during training.
@article{arxiv.2603.00054,
title = {Expert Divergence Learning for MoE-based Language Models},
author = {Jiaang Li and Haibin Chen and Langming Liu and Yujin Yuan and Yadao Wang and Yizhen Zhang and Chengting Yu and Xin Tong and Weidong Zhang and Shilei Liu and Wenbo Su and Bo Zheng},
journal= {arXiv preprint arXiv:2603.00054},
year = {2026}
}