CRASS:用于测试大语言模型反事实推理的新数据集与基准
计算与语言
2022-10-06 v3
摘要
我们引入了CRASS(反事实推理评估)数据集与基准,利用问题化反事实条件句作为评估大语言模型的新颖且有力的工具。我们介绍了该数据集设计及支持对照众包验证人类基线的评分基准。我们针对本基准测试了六个最先进的模型。结果显示其对这些模型构成了有效挑战,并为它们的改进留下了可观空间。
引用
@article{arxiv.2112.11941,
title = {CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models},
author = {Jörg Frohberg and Frank Binder},
journal= {arXiv preprint arXiv:2112.11941},
year = {2022}
}
备注
10 pages including references, plus 5 pages appendix. Edits for version 3 vs LREC 2022: Point out human baseline in abstract (also to match arxiv abstract), fix affiliation apergo.ai, and fix a recurring typo