Step-3 规模庞大且成本可控：面向高性价比解码的模型-系统协同设计
机器学习
2025-07-28 v1 人工智能
摘要
大型语言模型 (LLM) 在解码过程中面临硬件效率低下的问题，尤其是在长上下文推理任务中。本文介绍了 Step-3，一个拥有 3210 亿参数的视觉语言模型 (VLM)，其采用硬件感知的模型-系统协同设计，旨在最小化解码成本。Step-3 在两个关键维度进行了创新：(1) 一种新型多矩阵分解注意力 (MFA) 机制，在保持高注意力表达能力的同时，显著减小了 KV 缓存大小和计算量；(2) 注意力-前馈网络解耦 (AFD)，这是一种将注意力和前馈网络 (FFN) 层解耦为专用子系统的分布式推理系统。这种协同设计实现了前所未有的成本效益：与 DeepSeek-V3 和 Qwen3 MoE 235B 等模型相比，Step-3 显著降低了理论解码成本，且在更长上下文下优势进一步扩大。Step-3 在每个 token 激活 380 亿参数（多于 DeepSeek-V3 和 Qwen3 MoE 235B）的情况下实现了低成本，这表明硬件对齐的注意力算术强度、MoE 稀疏性和 AFD 对于成本效益至关重要。我们在 DeepSeek-V3 的优势场景下进行了直接对比。我们在 Hopper GPU 上的实现在 50ms TPOT SLA（4K 上下文，FP8，无 MTP）下实现了每 GPU 每秒高达 4,039 个 token 的解码吞吐量。这高于 DeepSeek-V3 在相同设置下的 2,324，并为 LLM 解码设定了新的帕累托前沿。
引用
@article{arxiv.2507.19427,
title = {Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding},
author = {StepFun and : and Bin Wang and Bojun Wang and Changyi Wan and Guanzhe Huang and Hanpeng Hu and Haonan Jia and Hao Nie and Mingliang Li and Nuo Chen and Siyu Chen and Song Yuan and Wuxun Xie and Xiaoniu Song and Xing Chen and Xingping Yang and Xuelin Zhang and Yanbo Yu and Yaoyu Wang and Yibo Zhu and Yimin Jiang and Yu Zhou and Yuanwei Lu and Houyi Li and Jingcheng Hu and Ka Man Lo and Ailin Huang and Binxing Jiao and Bo Li and Boyu Chen and Changxin Miao and Chang Lou and Chen Hu and Chen Xu and Chenfeng Yu and Chengyuan Yao and Daokuan Lv and Dapeng Shi and Deshan Sun and Ding Huang and Dingyuan Hu and Dongqing Pang and Enle Liu and Fajie Zhang and Fanqi Wan and Gulin Yan and Han Zhang and Han Zhou and Hanghao Wu and Hangyu Guo and Hanqi Chen and Hanshan Zhang and Hao Wu and Haocheng Zhang and Haolong Yan and Haoran Lv and Haoran Wei and Hebin Zhou and Heng Wang and Heng Wang and Hongxin Li and Hongyu Zhou and Hongyuan Wang and Huiyong Guo and Jia Wang and Jiahao Gong and Jialing Xie and Jian Zhou and Jianjian Sun and Jiaoren Wu and Jiaran Zhang and Jiayu Liu and Jie Cheng and Jie Luo and Jie Yan and Jie Yang and Jieyi Hou and Jinguang Zhang and Jinlan Cao and Jisheng Yin and Junfeng Liu and Junhao Huang and Junzhe Lin and Kaijun Tan and Kaixiang Li and Kang An and Kangheng Lin and Kenkun Liu and Lei Yang and Liang Zhao and Liangyu Chen and Lieyu Shi and Liguo Tan and Lin Lin and Lin Zhang and Lina Chen and Liwen Huang and Liying Shi and Longlong Gu and Mei Chen and Mengqiang Ren and Ming Li and Mingzhe Chen and Na Wang and Nan Wu and Qi Han and Qian Zhao and Qiang Zhang and Qianni Liu and Qiaohui Chen and Qiling Wu and Qinglin He and Qinyuan Tan and Qiufeng Wang and Qiuping Wu and Qiuyan Liang and Quan Sun and Rui Li and Ruihang Miao and Ruosi Wan and Ruyan Guo and Shangwu Zhong and Shaoliang Pang and Shengjie Fan and Shijie Shang and Shilei Jiang and Shiliang Yang and Shiming Hao and Shuli Gao and Siming Huang and Siqi Liu and Tiancheng Cao and Tianhao Cheng and Tianhao Peng and Wang You and Wei Ji and Wen Sun and Wenjin Deng and Wenqing He and Wenzhen Zheng and Xi Chen and Xiangwen Kong and Xianzhen Luo and Xiaobo Yang and Xiaojia Liu and Xiaoxiao Ren and Xin Han and Xin Li and Xin Wu and Xu Zhao and Yanan Wei and Yang Li and Yangguang Li and Yangshijie Xu and Yanming Xu and Yaqiang Shi and Yeqing Shen and Yi Yang and Yifei Yang and Yifeng Gong and Yihan Chen and Yijing Yang and Yinmin Zhang and Yizhuang Zhou and Yuanhao Ding and Yuantao Fan and Yuanzhen Yang and Yuchu Luo and Yue Peng and Yufan Lu and Yuhang Deng and Yuhe Yin and Yujie Liu and Yukun Chen and Yuling Zhao and Yun Mou and Yunlong Li and Yunzhou Ju and Yusheng Li and Yuxiang Yang and Yuxiang Zhang and Yuyang Chen and Zejia Weng and Zhe Xie and Zheng Ge and Zheng Gong and Zhenyi Lu and Zhewei Huang and Zhichao Chang and Zhiguo Huang and Zhirui Wang and Zidong Yang and Zili Wang and Ziqi Wang and Zixin Zhang and Binxing Jiao and Daxin Jiang and Heung-Yeung Shum and Xiangyu Zhang},
journal= {arXiv preprint arXiv:2507.19427},
year = {2025}
}