Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
Abstract
This technical report presents quantization-aware distillation (QAD) and our best practices for recovering accuracy of NVFP4-quantized large language models (LLMs) and vision-language models (VLMs). QAD distills a full-precision teacher model into a quantized student model using a KL divergence loss. While applying distillation to quantized models is not a new idea, we observe key advantages of QAD for today's LLMs: 1. It shows remarkable effectiveness and stability for models trained through multi-stage post-training pipelines, including supervised fine-tuning (SFT), reinforcement learning (RL), and model merging, where traditional quantization-aware training (QAT) suffers from engineering complexity and training instability; 2. It is robust to data quality and coverage, enabling accuracy recovery without full training data. We evaluate QAD across multiple post-trained models including AceReason Nemotron, Nemotron 3 Nano, Nemotron Nano V2, Nemotron Nano V2 VL (VLM), and Llama Nemotron Super v1, showing consistent recovery to near-BF16 accuracy.
Keywords
Cite
@article{arxiv.2601.20088,
title = {Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery},
author = {Meng Xin and Sweta Priyadarshi and Jingyu Xin and Bilal Kartal and Aditya Vavre and Asma Kuriparambil Thekkumpate and Zijia Chen and Ameya Sunil Mahabaleshwarkar and Ido Shahaf and Akhiad Bercovich and Kinjal Patel and Suguna Varshini Velury and Chenjie Luo and Zhiyu Cheng and Jenny Chen and Chen-Han Yu and Wei Ping and Oleg Rybakov and Nima Tajbakhsh and Oluwatobi Olabiyi and Dusan Stosic and Di Wu and Song Han and Eric Chung and Sharath Turuvekere Sreenivas and Bryan Catanzaro and Yoshi Suhara and Tijmen Blankevoort and Huizi Mao},
journal= {arXiv preprint arXiv:2601.20088},
year = {2026}
}