Dist2ill: 用于大型语言模型单次推断分布性蒸馏不确定性估计
机器学习
2026-02-10 v3 人工智能
计算与语言
摘要
大型语言模型(LLM)常常在生成响应的质量与其赋予的置信度估计之间存在错位。贝叶斯方法，如对可靠权重后验或推理路径空间进行边际化，提供了有效的解决方案，但由于在测试时需要重复抽样，计算开销较大。为实现单次前向传播中的准确不确定性估计，我们提出一种新颖的分布性蒸馏框架(Dist2ill)，该框架训练LLM在一次推断内产生多个多样化的推理路径，同时使用轻量级参数化模块近似来自抽样分布的经验置信度得分。广泛的实验表明，Dist2ill保持推理多样性，实现了状态最好的不确定性估计，显著改善了Expected Calibration Error(ECE)和Negative Log-Likelihood(NLL)，同时保持计算效率。
引用
@article{arxiv.2505.11731,
title = {Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models},
author = {Yicong Zhao and King Yeung Tsang and Harshil Vejendla and Haizhou Shi and Zhuohang Li and Zhigang Hua and Qi Xu and Tunyu Zhang and Yi Wang and Ligong Han and Bradley A. Malin and Hao Wang},
journal= {arXiv preprint arXiv:2505.11731},
year = {2026}
}
备注
Preprint; work in progress. Update Log: 05/2025 (v1&v2): Introduced Dist2ill (previously named EUD) for efficient uncertainty estimation, focusing on discriminative reasoning tasks. 02/2026 (v3): Extended Dist2ill to a unified framework supporting both discriminative and generative reasoning