使用 NVFP4 预训练大型语言模型
计算与语言
2026-03-06 v2 人工智能
机器学习
摘要
如今，大型语言模型（LLM）是跨许多领域的强大问题求解器，并且随着模型规模、训练集规模和训练集质量的扩大，它们继续变得更加强大，正如全行业广泛的研究和实验所表明的那样。如今训练一个前沿模型需要大约数十到数百 yottaflops 的计算量，这是时间、计算和能源的巨大投资。因此，提高预训练效率对于实现下一代更强大的 LLM 至关重要。虽然 8 位浮点（FP8）训练现在已被广泛采用，但过渡到更窄的精度（如 4 位浮点，FP4）可以进一步提升计算速度和资源利用率。然而，这一级别的量化对训练稳定性、收敛性和实现提出了挑战，特别是对于在长 token 跨度上训练的大规模模型。在本研究中，我们引入了一种使用 NVFP4 格式对大型语言模型（LLM）进行稳定且精确训练的新方法。我们的方法集成了随机 Hadamard 变换（RHT）来限制块级异常值，采用二维量化方案以在正向和反向传播中保持一致的表示，利用随机舍入进行无偏梯度估计，并结合了选择性高精度层。我们通过在 10 万亿个 token 上训练一个 120 亿参数的模型来验证我们的方法——这是迄今为止在 4 位精度下最长且有公开记录的训练运行。我们的结果表明，使用我们基于 NVFP4 的预训练技术训练的模型实现了与 FP8 基线相当的训练损失和下游任务准确率。这些发现表明，NVFP4 与我们的训练方法相结合，代表了窄精度 LLM 训练算法向前迈出的重要一步。
引用
@article{arxiv.2509.25149,
title = {Pretraining Large Language Models with NVFP4},
author = {NVIDIA and Felix Abecassis and Anjulie Agrusa and Dong Ahn and Jonah Alben and Stefania Alborghetti and Michael Andersch and Sivakumar Arayandi and Alexis Bjorlin and Aaron Blakeman and Evan Briones and Ian Buck and Bryan Catanzaro and Muya Chang and Jinhang Choi and Mike Chrzanowski and Eric Chung and Victor Cui and Steve Dai and Bita Darvish Rouhani and Carlo del Mundo and Deena Donia and Burc Eryilmaz and Henry Estela and Abhinav Goel and Oleg Goncharov and Yugi Guvvala and Robert Hesse and Russell Hewett and Herbert Hum and Ujval Kapasi and Brucek Khailany and Mikail Khona and Nick Knight and Alex Kondratenko and Ronny Krashinsky and Ben Lanir and Simon Layton and Michael Lightstone and Daniel Lo and Paulius Micikevicius and Asit Mishra and Tim Moon and Deepak Narayanan and Chao Ni and Abhijit Paithankar and Satish Pasumarthi and Ankit Patel and Mostofa Patwary and Ashwin Poojary and Gargi Prasad and Sweta Priyadarshi and Yigong Qin and Xiaowei Ren and Oleg Rybakov and Charbel Sakr and Sanjeev Satheesh and Stas Sergienko and Pasha Shamis and Kirthi Shankar and Nishant Sharma and Mohammad Shoeybi and Michael Siu and Misha Smelyanskiy and Darko Stosic and Dusan Stosic and Bor-Yiing Su and Frank Sun and Nima Tajbakhsh and Shelby Thomas and Przemek Tredak and Evgeny Tsykunov and Gandhi Vaithilingam and Aditya Vavre and Rangharajan Venkatesan and Roger Waleffe and Qiyu Wan and Hexin Wang and Mengdi Wang and Lizzie Wei and Hao Wu and Evan Wu and Keith Wyss and Ning Xu and Jinze Xue and Charlene Yang and Yujia Zhai and Ruoxi Zhang and Jingyang Zhu and Zhongbo Zhu},
journal= {arXiv preprint arXiv:2509.25149},
year = {2026}
}
备注
Update includes: (1) fixing a typo in eq. 2 (2) updating author list, and (3) adding a related work