CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

Jörg Frohberg; Frank Binder

CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

Computation and Language 2022-10-06 v3

Authors: Jörg Frohberg , Frank Binder

Abstract

We introduce the CRASS (counterfactual reasoning assessment) data set and benchmark utilizing questionized counterfactual conditionals as a novel and powerful tool to evaluate large language models. We present the data set design and benchmark that supports scoring against a crowd-validated human baseline. We test six state-of-the-art models against our benchmark. Our results show that it poses a valid challenge for these models and opens up considerable room for their improvement.

Keywords

counterfactual reasoning large language models benchmark crass dataset questionized counterfactuals model evaluation

Cite

@article{arxiv.2112.11941,
  title  = {CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models},
  author = {Jörg Frohberg and Frank Binder},
  journal= {arXiv preprint arXiv:2112.11941},
  year   = {2022}
}

Comments

10 pages including references, plus 5 pages appendix. Edits for version 3 vs LREC 2022: Point out human baseline in abstract (also to match arxiv abstract), fix affiliation apergo.ai, and fix a recurring typo

CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models

Abstract

Keywords

Cite

Comments

Related papers