Pretraining Large Language Models with NVFP4
Abstract
Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as shown by extensive research and experimentation across the industry. Training a frontier model today requires on the order of tens to hundreds of yottaflops, which is a massive investment of time, compute, and energy. Improving pretraining efficiency is therefore essential to enable the next generation of even more capable LLMs. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision, such as 4-bit floating point (FP4), could unlock additional improvements in computational speed and resource utilization. However, quantization at this level poses challenges to training stability, convergence, and implementation, notably for large-scale models trained on long token horizons. In this study, we introduce a novel approach for stable and accurate training of large language models (LLMs) using the NVFP4 format. Our method integrates Random Hadamard transforms (RHT) to bound block-level outliers, employs a two-dimensional quantization scheme for consistent representations across both the forward and backward passes, utilizes stochastic rounding for unbiased gradient estimation, and incorporates selective high-precision layers. We validate our approach by training a 12-billion-parameter model on 10 trillion tokens -- the longest publicly documented training run in 4-bit precision to date. Our results show that the model trained with our NVFP4-based pretraining technique achieves training loss and downstream task accuracies comparable to an FP8 baseline. These findings highlight that NVFP4, when combined with our training approach, represents a major step forward in narrow-precision LLM training algorithms.
Cite
@article{arxiv.2509.25149,
title = {Pretraining Large Language Models with NVFP4},
author = {NVIDIA and Felix Abecassis and Anjulie Agrusa and Dong Ahn and Jonah Alben and Stefania Alborghetti and Michael Andersch and Sivakumar Arayandi and Alexis Bjorlin and Aaron Blakeman and Evan Briones and Ian Buck and Bryan Catanzaro and Muya Chang and Jinhang Choi and Mike Chrzanowski and Eric Chung and Victor Cui and Steve Dai and Bita Darvish Rouhani and Carlo del Mundo and Deena Donia and Burc Eryilmaz and Henry Estela and Abhinav Goel and Oleg Goncharov and Yugi Guvvala and Robert Hesse and Russell Hewett and Herbert Hum and Ujval Kapasi and Brucek Khailany and Mikail Khona and Nick Knight and Alex Kondratenko and Ronny Krashinsky and Ben Lanir and Simon Layton and Michael Lightstone and Daniel Lo and Paulius Micikevicius and Asit Mishra and Tim Moon and Deepak Narayanan and Chao Ni and Abhijit Paithankar and Satish Pasumarthi and Ankit Patel and Mostofa Patwary and Ashwin Poojary and Gargi Prasad and Sweta Priyadarshi and Yigong Qin and Xiaowei Ren and Oleg Rybakov and Charbel Sakr and Sanjeev Satheesh and Stas Sergienko and Pasha Shamis and Kirthi Shankar and Nishant Sharma and Mohammad Shoeybi and Michael Siu and Misha Smelyanskiy and Darko Stosic and Dusan Stosic and Bor-Yiing Su and Frank Sun and Nima Tajbakhsh and Shelby Thomas and Przemek Tredak and Evgeny Tsykunov and Gandhi Vaithilingam and Aditya Vavre and Rangharajan Venkatesan and Roger Waleffe and Qiyu Wan and Hexin Wang and Mengdi Wang and Lizzie Wei and Hao Wu and Evan Wu and Keith Wyss and Ning Xu and Jinze Xue and Charlene Yang and Yujia Zhai and Ruoxi Zhang and Jingyang Zhu and Zhongbo Zhu},
journal= {arXiv preprint arXiv:2509.25149},
year = {2026}
}
Comments
Update includes: (1) fixing a typo in eq. 2 (2) updating author list, and (3) adding a related work