English

One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence

Artificial Intelligence 2026-07-30 v1

Abstract

A single human must audit NN LLM agents under a budget of BNB \ll N audits per round, guided by self-reported confidence that may be adversarially miscalibrated and by correlated errors. We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold δ\delta^* past which confidence-ranked auditing is \emph{worse} than random. Two a-priori expectations reverse: δ\delta^* \emph{rises} as the budget shrinks, and cross-family correlation is not low---shared difficulty dominates lineage. Five open-weight LLMs show operationally useless (near-constant) confidence, point estimates at or beyond the flip though CIs straddle it; a proprietary model is informative and lands below it. We give a quantitative criterion for \emph{vacuous} oversight, and replaying policies on recorded traces confirms the ordering.

Keywords

Cite

@article{arxiv.2607.28317,
  title  = {One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence},
  author = {Cesare Zavattari and Alessandro Tommasi and Giuseppe Prencipe},
  journal= {arXiv preprint arXiv:2607.28317},
  year   = {2026}
}