English

Can Chat GPT solve a Linguistics Exam?

Computation and Language 2023-11-07 v1

Abstract

The present study asks if ChatGPT4, the version of ChatGPT which uses the language model GPT4, can successfully solve introductory linguistic exams. Previous exam questions of an Introduction to Linguistics course at a German university are used to test this. The exam questions were fed into ChatGPT4 with only minimal preprocessing. The results show that the language model is very successful in the interpretation even of complex and nested tasks. It proved surprisingly successful in the task of broad phonetic transcription, but performed less well in the analysis of morphemes and phrases. In simple cases it performs sufficiently well, but rarer cases, particularly with missing one-to-one correspondence, are currently treated with mixed results. The model is not yet able to deal with visualisations, such as the analysis or generation of syntax trees. More extensive preprocessing, which translates these tasks into text data, allow the model to also solve these tasks successfully.

Keywords

Cite

@article{arxiv.2311.02499,
  title  = {Can Chat GPT solve a Linguistics Exam?},
  author = {Patricia Ronan and Gerold Schneider},
  journal= {arXiv preprint arXiv:2311.02499},
  year   = {2023}
}

Comments

13 pages, 7 figures, presentation at ISLE (International Society for the Linguistics of English) 7, 2023, Brisbane

R2 v1 2026-06-28T13:11:42.881Z