MiniMax-01：基于闪电注意力扩展基础模型
计算与语言
2025-01-15 v1 计算机视觉与模式识别
摘要
我们介绍了MiniMax-01系列，包括MiniMax-Text-01和MiniMax-VL-01，这些模型在性能上与顶级模型相当，同时在处理更长上下文方面具有优势。其核心在于闪电注意力及其高效扩展。为最大化计算容量，我们将其与混合专家(MoE)集成，创建了一个拥有32个专家、总参数数为4560亿的参数模型，其中每次标记激活459亿参数。我们开发了针对MoE和闪电注意力的优化并行策略和高度高效的计算-通信重叠技术。这种方法使我们能够在上下文跨越数百万标记的模型上进行高效训练和推理。MiniMax-Text-01的上下文窗口在训练时可达100万标记，推理时可 extrapolate至400万标记且成本可接受。我们的视觉-语言模型MiniMax-VL-01通过使用5120亿个视觉-语言标记进行持续训练而构建。在针对标准和内部基准的实验中，我们的模型在性能上与GPT-4o和Claude-3.5-Sonnet等最先进模型相当，同时提供了20-32倍更长的上下文窗口。我们在https://github.com/MiniMax-AI上公开发布了MiniMax-01。
引用
@article{arxiv.2501.08313,
title = {MiniMax-01: Scaling Foundation Models with Lightning Attention},
author = {MiniMax and Aonian Li and Bangwei Gong and Bo Yang and Boji Shan and Chang Liu and Cheng Zhu and Chunhao Zhang and Congchao Guo and Da Chen and Dong Li and Enwei Jiao and Gengxin Li and Guojun Zhang and Haohai Sun and Houze Dong and Jiadai Zhu and Jiaqi Zhuang and Jiayuan Song and Jin Zhu and Jingtao Han and Jingyang Li and Junbin Xie and Junhao Xu and Junjie Yan and Kaishun Zhang and Kecheng Xiao and Kexi Kang and Le Han and Leyang Wang and Lianfei Yu and Liheng Feng and Lin Zheng and Linbo Chai and Long Xing and Meizhi Ju and Mingyuan Chi and Mozhi Zhang and Peikai Huang and Pengcheng Niu and Pengfei Li and Pengyu Zhao and Qi Yang and Qidi Xu and Qiexiang Wang and Qin Wang and Qiuhui Li and Ruitao Leng and Shengmin Shi and Shuqi Yu and Sichen Li and Songquan Zhu and Tao Huang and Tianrun Liang and Weigao Sun and Weixuan Sun and Weiyu Cheng and Wenkai Li and Xiangjun Song and Xiao Su and Xiaodong Han and Xinjie Zhang and Xinzhu Hou and Xu Min and Xun Zou and Xuyang Shen and Yan Gong and Yingjie Zhu and Yipeng Zhou and Yiran Zhong and Yongyi Hu and Yuanxiang Fan and Yue Yu and Yufeng Yang and Yuhao Li and Yunan Huang and Yunji Li and Yunpeng Huang and Yunzhi Xu and Yuxin Mao and Zehan Li and Zekang Li and Zewei Tao and Zewen Ying and Zhaoyang Cong and Zhen Qin and Zhenhua Fan and Zhihang Yu and Zhuo Jiang and Zijia Wu},
journal= {arXiv preprint arXiv:2501.08313},
year = {2025}
}
备注
A technical report from MiniMax. The authors are listed in alphabetical order. We open-sourced our MiniMax-01 at https://github.com/MiniMax-AI