Examining the Emergence of Deductive Reasoning in Generative Language Models
Computation and Language
2023-06-05 v1 Artificial Intelligence
Machine Learning
Abstract
We conduct a preliminary inquiry into the ability of generative transformer models to deductively reason from premises provided. We observe notable differences in the performance of models coming from different training setups and find that the deductive reasoning ability increases with scale. Further, we discover that the performance generally does not decrease with the length of the deductive chain needed to reach the conclusion, with the exception of OpenAI GPT-3 and GPT-3.5 models. Our study considers a wide variety of transformer-decoder models, ranging from 117 million to 175 billion parameters in size.
Cite
@article{arxiv.2306.01009,
title = {Examining the Emergence of Deductive Reasoning in Generative Language Models},
author = {Peter Belcak and Luca A. Lanzendörfer and Roger Wattenhofer},
journal= {arXiv preprint arXiv:2306.01009},
year = {2023}
}
Comments
Accepted to the 1st Natural Language Reasoning and Structured Explanations Workshop (NLRSE@ACL'23). 8 pages, 4 figures, 3 tables