提升 LLM-as-a-Judge 能力：一种通用能力提升方法
计算与语言
2025-09-09 v2
摘要
LLM-as-Judge 利用大语言模型（LLM）的生成和推理能力，对LLM响应在多样化情景下的表现进行评估，提供准确的偏好信号。这一方法在确保AI输出符合人类价值观、符合社会规范方面发挥着关键作用。近期研究提出了许多方法来训练LLM作为评判者，但这些方法大多数据需求高或缺乏准确性，且仅关注LLM的评判能力。本文将评判能力视为LLM的一种通用能力，通过包含监督微调（SFT）预热和直接偏好优化（DPO）增强的两阶段训练方法，实现评判风格适应并提高判断准确性。此外，我们引入一种高效的数据合成方法来生成评判内容。实验结果表明，我们的方法仅需约占其他方法所需数据的2%至40%，在RewardBench上实现SOTA性能。 Furthermore, our training method enhances the general capabilities of the model by constructing complicated judge task, and the judge signals provided by our model have significantly enhanced the downstream DPO training performance of our internal models in our test to optimize policy model with Judge Model. We also open-source our model weights and training data to facilitate further research.
关键词
引用
@article{arxiv.2502.11689,
title = {Improve LLM-as-a-Judge Ability as a General Ability},
author = {Jiachen Yu and Shaoning Sun and Xiaohui Hu and Jiaxu Yan and Kaidong Yu and Xuelong Li},
journal= {arXiv preprint arXiv:2502.11689},
year = {2025}
}