将合成数据评估 grounding 到无监督文档语料库中的语言模型
摘要
语言模型（LM）持续进步，提升了响应质量和连贯性。鉴于Internet规模的训练数据集，LM在训练期间可能已遇到用户可能要求其生成的大量内容。已构建了大量评估基准来衡量模型质量、响应合适性和推理能力。然而，人力 effort for benchmark construction is rapidly being outpaced by the size and scope of the models under evaluation. Having humans build a benchmark for every possible domain of interest is impractical. Therefore, we propose a methodology for automating the construction of fact-based synthetic data model evaluations grounded in document populations. This work leverages the same LMs to evaluate domain-specific knowledge automatically, using only grounding documents (e.g., a textbook) as input. This generative benchmarking approach corresponds well with human curated questions producing an ensemble Spearman ranking correlation of and a benchmark evaluation Pearson accuracy correlation of (model specific ). This novel approach supports generating both multiple choice and open-ended synthetic data questions to gain diagnostic insight of LM capability. We apply this methodology to evaluate model performance on three recent documents (two post LM knowledge cutoff), discovering a surprisingly strong performance from Gemma-3 models on open-ended questions. Code is available at https://github.com/mmajurski/grounded-synth-lm-benchmark
引用
@article{arxiv.2505.08905,
title = {Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora},
author = {Michael Majurski and Cynthia Matuszek},
journal= {arXiv preprint arXiv:2505.08905},
year = {2026}
}