大型语言模型的元评判：概念、方法与挑战
摘要
大型语言模型(LLMs)正在快速发展，如今经常被用作评估器，通常称为 LLM-as-a-Judge 的过程，提供模型输出的质量评估。然而，近期研究指出，这种评估存在显著漏洞，包括对提示敏感、系统性偏见、verbosity effects以及不可靠或幻觉的理由。这些限制促使开发更健壮的范式，称为 LLM-as-a-Meta-Judge。本 survey reviews recent advances in meta-judging and organizes the literature, by introducing a framework along six key perspectives: (i) Conceptual Foundations, (ii) Mechanisms of Meta-Judging, (iii) Alignment Training Methods, (iv) Evaluation, (v) Limitations and Failure Modes, and (vi) Future Directions. By analyzing the limitations of LLM-as-a-Judge and summarizing recent advances in meta-judging by LLMs, we argue that LLM-as-a-Meta-Judge offers a promising direction for more stable and trustworthy automated evaluation, while highlighting remaining challenges related to cost, prompt sensitivity, and shared model biases, which must be addressed to advance the next generation of LLM evaluation methodologies.
引用
@article{arxiv.2601.17312,
title = {Meta-Judging with Large Language Models: Concepts, Methods, and Challenges},
author = {Hugo Silva and Mateus Mendes and Hugo Gonçalo Oliveira},
journal= {arXiv preprint arXiv:2601.17312},
year = {2026}
}