JoyAI-LLM Flash：通过标记效率推进中等规模大语言模型
计算与语言
2026-04-09 v2 人工智能
摘要
我们提出了 JoyAI-LLM Flash，一种高效的混合专家（MoE）语言模型，旨在重新定义子 50B 参数规模下强性能与标记效率之间的权衡。JoyAI-LLM Flash 在包含 20 万亿标记的庞大语料库上进行预训练，并通过严格的 post-training 流水线进一步优化，包括监督微调（SFT）、直接偏好优化（DPO）以及跨多种环境的大规模强化学习（RL）。为提高标记效率，JoyAI-LLM Flash 在思考模式与非思考模式之间进行了战略性平衡，并引入了 FiberPO，这是一种受纤维化理论启发的新 RL 算法，将信任域维护分解为全局和局部组件，为 LLM 策略优化提供统一的多尺度稳定性控制。为增强架构稀疏性，该模型包含 48B 总参数，但每次前向传播仅激活 2.7B 参数，实现了比同类规模行业领先模型高得多的稀疏比。为进一步提高推理吞吐量，我们采用了联合训练-推理协同设计，集成了密集多标记预测（MTP）和量化感知训练（QAT）。我们在 Hugging Face 上发布了 JoyAI-LLM-48B-A3B Base 及其实例化变体的检查点，以支持开源社区。
引用
@article{arxiv.2604.03044,
title = {JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency},
author = {Aichen Cai and Anmeng Zhang and Anyu Li and Bo Zhang and Bohua Cai and Chang Li and Changjian Jiang and Changkai Lu and Chao Xue and Chaocai Liang and Cheng Zhang and Dongkai Liu and Fei Wang and Guoqiang Huang and Haijian Ke and Han Lin and Hao Wang and Ji Miao and Jiacheng Zhang and Jialong Shi and Jifeng Zhu and Jingjing Qian and Junhui Luo and Junwu Xiong and Lam So and Liang Huang and Ming Ke and Mingyang Li and Panfeng Shi and Peng Hao and Qi Wang and Qian Lai and Qiaoqiao Yuan and Qingyu Yin and Qiong Cao and Qixiang Wang and Rongcheng Bian and Rongduo Han and Shaoqiang Zheng and Shi Hu and Shi Suo and Shijie Ren and Shijin Zhang and Shiying Fan and Shuai Xie and Tianyi Zhang and Wei Liu and Wentao Tan and Xianghan Meng and Xiaodong He and Xing Pan and Xiran Wang and Xuyang Peng and Ya Zhang and Yang Liu and Yangyang Duan and Yanxu Chen and Yicheng Gong and Yidan Huang and Yifei Liu and Yinhao Bai and Yongqiang Liu and Yuesong Zhang and Yuqi Zhang and Zerui Xie and Zhenfang Wang and Zhennan Shen and Zheyuan Liu and Zhuwei Zeng},
journal= {arXiv preprint arXiv:2604.03044},
year = {2026}
}
备注
Xiaodong He is the corresponding author