Hunyuan-Large：腾讯开源的 520 亿激活参数 MoE 模型
计算与语言
2024-11-07 v3 人工智能
摘要
在本文中，我们介绍了 Hunyuan-Large，这是目前最大的开源基于 Transformer 的混合专家模型，总参数量为 3890 亿，激活参数量为 520 亿，能够处理长达 256K 的令牌。我们在各种基准测试上对 Hunyuan-Large 的优越性能进行了全面评估，包括语言理解与生成、逻辑推理、数学问题求解、编程、长上下文和聚合任务，其中它超越了 LLama3.1-70B，并在与显著更大的 LLama3.1-405B 模型比较时表现出可比的性能。Hunyuan-Large 的关键实践包括比先前文献大数个数量级的大规模合成数据、混合专家路由策略、键值缓存压缩技术以及专家专用学习率策略。此外，我们还研究了混合专家模型的缩放定律和学习率调度，为未来的模型开发与优化提供了有价值的见解和指导。Hunyuan-Large 的代码和检查点已发布，以促进未来的创新与应用。
引用
@article{arxiv.2411.02265,
title = {Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent},
author = {Xingwu Sun and Yanfeng Chen and Yiqing Huang and Ruobing Xie and Jiaqi Zhu and Kai Zhang and Shuaipeng Li and Zhen Yang and Jonny Han and Xiaobo Shu and Jiahao Bu and Zhongzhi Chen and Xuemeng Huang and Fengzong Lian and Saiyong Yang and Jianfeng Yan and Yuyuan Zeng and Xiaoqin Ren and Chao Yu and Lulu Wu and Yue Mao and Jun Xia and Tao Yang and Suncong Zheng and Kan Wu and Dian Jiao and Jinbao Xue and Xipeng Zhang and Decheng Wu and Kai Liu and Dengpeng Wu and Guanghui Xu and Shaohua Chen and Shuang Chen and Xiao Feng and Yigeng Hong and Junqiang Zheng and Chengcheng Xu and Zongwei Li and Xiong Kuang and Jianglu Hu and Yiqi Chen and Yuchi Deng and Guiyang Li and Ao Liu and Chenchen Zhang and Shihui Hu and Zilong Zhao and Zifan Wu and Yao Ding and Weichao Wang and Han Liu and Roberts Wang and Hao Fei and Peijie Yu and Ze Zhao and Xun Cao and Hai Wang and Fusheng Xiang and Mengyuan Huang and Zhiyuan Xiong and Bin Hu and Xuebin Hou and Lei Jiang and Jianqiang Ma and Jiajia Wu and Yaping Deng and Yi Shen and Qian Wang and Weijie Liu and Jie Liu and Meng Chen and Liang Dong and Weiwen Jia and Hu Chen and Feifei Liu and Rui Yuan and Huilin Xu and Zhenxiang Yan and Tengfei Cao and Zhichao Hu and Xinhua Feng and Dong Du and Tinghao Yu and Yangyu Tao and Feng Zhang and Jianchen Zhu and Chengzhong Xu and Xirui Li and Chong Zha and Wen Ouyang and Yinben Xia and Xiang Li and Zekun He and Rongpeng Chen and Jiawei Song and Ruibin Chen and Fan Jiang and Chongqing Zhao and Bo Wang and Hao Gong and Rong Gan and Winston Hu and Zhanhui Kang and Yong Yang and Yuhong Liu and Di Wang and Jie Jiang},
journal= {arXiv preprint arXiv:2411.02265},
year = {2024}
}
备注
17 pages, 4 Figures