Large language models (LLMs) are now widely used as judges, yet their decisions can change under presentation choices that should be irrelevant. We study one such source of instability: candidate-order sensitivity in listwise factuality evaluation, where several answers can look similarly polished while differing substantially in hallucination risk. We introduce PCFJudge, an inference-time method that reruns the same factuality-first listwise prompt over multiple orderings of the same candidate set and aggregates the resulting scores, ranks, and uncertainty signals into a single consensus decision. On RewardBench 2 Factuality, the final seven-permutation aggregate (K=7) improves top-1 selection accuracy from 86.00% to 91.33% with GPT-5.4 and from 86.33% to 89.67% with Claude Sonnet 4.6. These results suggest that candidate order can be a meaningful source of factuality-judging error and that marginalizing over this nuisance variation can improve the reliability of LLM evaluation.
@article{arxiv.2603.20562,
title = {Permutation-Consensus Listwise Judging for Robust Factuality Evaluation},
author = {Tianyi Huang and Nathan Huang and Justin Tang and Wenqian Chen and Elsa Fan},
journal= {arXiv preprint arXiv:2603.20562},
year = {2026}
}
Comments
Accepted at the Fifth Workshop on Natural Language Generation, Evaluation, and Metrics at ACL 2026