Arcee Trinity 大型技术报告
机器学习
2026-02-20 v1 计算与语言
摘要
我们报告Arcee Trinity Large的技术细节，这是一个稀疏Mixture-of-Experts模型，总参数为400B，每token激活13B参数。此外，我们报告Trinity Nano和Trinity Mini，Trinity Nano总参数为6B，每token激活1B参数，Trinity Mini总参数为26B，每token激活3B参数。该模型的现代架构包括交错的局部和全局注意力、门控注意力、深度比例 sandwich 规范以及用于Mixture-of-Experts的sigmoid路由。对于Trinity Large，我们还引入一种新的MoE负载平衡策略，称为Soft-clamped Momentum Expert Bias Updates (SMEBU)。我们采用Muon优化器训练这些模型。所有三个模型训练完成且无loss峰值。Trinity Nano和Trinity Mini在10万亿token上进行预训练，Trinity Large在17万亿token上进行预训练。模型checkpoint已在https://huggingface.co/arcee-ai/提供。
引用
@article{arxiv.2602.17004,
title = {Arcee Trinity Large Technical Report},
author = {Varun Singh and Lucas Krauss and Sami Jaghouar and Matej Sirovatka and Charles Goddard and Fares Obied and Jack Min Ong and Jannik Straube and Fern and Aria Harley and Conner Stewart and Colin Kealty and Maziyar Panahi and Simon Kirsten and Anushka Deshpande and Anneketh Vij and Arthur Bresnu and Pranav Veldurthi and Raghav Ravishankar and Hardik Bishnoi and DatologyAI Team and Arcee AI Team and Prime Intellect Team and Mark McQuade and Johannes Hagemann and Lucas Atkins},
journal= {arXiv preprint arXiv:2602.17004},
year = {2026}
}