Human-in-the-Loop Nugget Annotation for Accountable LLM-as-a-Judge Evaluations

Laura Dietz

Human-in-the-Loop Nugget Annotation for Accountable LLM-as-a-Judge Evaluations

Information Retrieval 2026-06-27 v1

Authors: Laura Dietz

Abstract

Evaluating AI/Agentic system outputs reliably requires human judgment, but how one incorporates the human determines whether one gets a real quality signal or expensive theater. The common approaches either accidentally anchor human experts (leading to rubber-stamping) or leave them unsupported in high-variance labeling tasks. We present a prototype annotation tool that implements a different division of labor: humans identify what information matters (nuggets), while LLMs handle high-volume matching of nuggets to system outputs. This plays to each party's strengths while maintaining genuine human oversight. We describe the three-phase workflow, key design decisions, and how exported nugget banks integrate with automated judges.

Keywords

human-in-the-loop nugget annotation LLM-as-a-judge evaluation methodology annotation tool accountability

Cite

@article{arxiv.2606.29033,
  title  = {Human-in-the-Loop Nugget Annotation for Accountable LLM-as-a-Judge Evaluations},
  author = {Laura Dietz},
  journal= {arXiv preprint arXiv:2606.29033},
  year   = {2026}
}