HomeMachine LearningarXiv:2605.29857

Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?

Machine Learning2026-05v1license

Abstract

Large language models (LLMs) are increasingly used for writing and review support, but their usefulness depends on context-dependent criteria, such as expert preferences or organization-specific conventions, that are often tacit, undocumented, and difficult to elicit directly. We propose a problem setting for learning reusable natural-language rubrics from accumulated inline comments on artifacts such as human-written or LLM-generated drafts. Our method infers rubrics from these comments and iteratively refines them by observing comment-wise mismatches between rubric-conditioned predictions and reference comments. We evaluate the proposed method in real-world review settings and in controlled settings with reference rubrics. These results show that inline comments can be distilled into reusable rubrics that support comment prediction, rubric understanding, and automatic artifact revision.

Cite

@article{arxiv.2605.29857,
  title  = {Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?},
  author = {Kotaro Yoshida and So Kuroki and Yuki Imajuku and Taishi Nakamura and Ryunosuke Iwai and Haruki Goda and Takuya Akiba},
  journal= {arXiv preprint arXiv:2605.29857},
  year   = {2026}
}