English

DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following

Computation and Language 2026-03-05 v1 Artificial Intelligence

Abstract

Evaluating instruction following in Large Language Models requires decomposing instructions into verifiable requirements and assessing satisfaction--tasks currently dependent on manual annotation and uniform criteria that do not align with human judgment patterns. We present DIALEVAL, a type-theoretic framework using dual LLM agents to automate instruction decomposition into typed predicates and implement type-specific satisfaction semantics. The framework enforces formal atomicity and independence constraints during automated extraction, then applies differentiated evaluation criteria--semantic equivalence for content predicates, exact precision for numerical predicates--mirroring empirically observed human assessment patterns. Extended to multi-turn dialogues through history-aware satisfaction functions, DIALEVAL enables evaluation in conversational contexts where single-turn methods fail. Validation demonstrates 90.38% accuracy (26.45% error reduction over baselines) and substantially stronger correlation with human judgment for complex instructions.

Keywords

Cite

@article{arxiv.2603.03321,
  title  = {DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following},
  author = {Nardine Basta and Dali Kaafar},
  journal= {arXiv preprint arXiv:2603.03321},
  year   = {2026}
}

Comments

PAKDD 2026