English

OpenAI Cribbed Our Tax Example, But Can GPT-4 Really Do Tax?

Artificial Intelligence 2024-02-08 v2 Computation and Language
Authors: Andrew Blair-Stanek , Nils Holzenberger , Benjamin Van Durme
View on arXiv ↗ PDF ↗

Abstract

The authors explain where OpenAI got the tax law example in its livestream demonstration of GPT-4, why GPT-4 got the wrong answer, and how it fails to reliably calculate taxes.

Keywords

gpt-4 tax law tax calculation large language model error analysis

Cite

@article{arxiv.2309.09992,
  title  = {OpenAI Cribbed Our Tax Example, But Can GPT-4 Really Do Tax?},
  author = {Andrew Blair-Stanek and Nils Holzenberger and Benjamin Van Durme},
  journal= {arXiv preprint arXiv:2309.09992},
  year   = {2024}
}

Comments

5 pages

Related papers

View all related →
Computation and Language · Computer Science
GPT-4 Can't Reason
Konstantine Arkoudas
2023-08-11
Artificial Intelligence · Computer Science
OpenAi's GPT4 as coding assistant
Lefteris Moussiades, George Zografos
2023-09-25
Machine Learning · Computer Science
Large Language Models' Understanding of Math: Source Criticism and Extrapolation
Roozbeh Yousefzadeh, Xuenan Cao
2023-11-15
Computers and Society · Computer Science
False Sense of Security in Explainable Artificial Intelligence (XAI)
Neo Christopher Chung, Hongkyou Chung, Hearim Lee, Lennart Brocki +2
2024-06-14
Artificial Intelligence · Computer Science
AI-assisted coding: Experiments with GPT-4
Russell A Poldrack, Thomas Lu, Gašper Beguš
2023-04-27
Artificial Intelligence · Computer Science
Argumentation-Based Explainability for Legal AI: Comparative and Regulatory Perspectives
Andrada Iulia Prajescu, Roberto Confalonieri
2025-10-14
Artificial Intelligence · Computer Science
The Opaque Law of Artificial Intelligence
Vincenzo Calderonio
2025-01-24
Computation and Language · Computer Science
Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing
Jawid Ahmad Baktash, Mursal Dawodi
2023-05-08
Artificial Intelligence · Computer Science
Evaluating AI for Law: Bridging the Gap with Open-Source Solutions
Rohan Bhambhoria, Samuel Dahan, Jonathan Li, Xiaodan Zhu
2024-04-19
Computation and Language · Computer Science
Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code
Jaromir Savelka, Arav Agarwal, Christopher Bogart, Majd Sakr
2023-03-15
Computer Vision and Pattern Recognition · Computer Science
A novel approach to generate datasets with XAI ground truth to evaluate image models
Miquel Miró-Nicolau, Antoni Jaume-i-Capó, Gabriel Moyà-Alcover
2023-10-05
Computers and Society · Computer Science
The AI Act proposal: a new right to technical interpretability?
Chiara Gallese
2023-03-31
Computation and Language · Computer Science
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke +10
2023-04-17
Software Engineering · Computer Science
An Empirical Study of OpenAI API Discussions on Stack Overflow
Xiang Chen, Jibin Wang, Chaoyang Gao, Xiaolin Ju +1
2025-05-08
Computers and Society · Computer Science
Algorithmic Governance for Explainability: A Comparative Overview of Progress and Trends
Yulu Pi
2023-03-02
Computation and Language · Computer Science
Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence
John J. Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao +5
2023-06-13
Computation and Language · Computer Science
Explaining Legal Concepts with Augmented Large Language Models (GPT-4)
Jaromir Savelka, Kevin D. Ashley, Morgan A. Gray, Hannes Westermann +1
2023-06-23
Human-Computer Interaction · Computer Science
Reimagining Legal Fact Verification with GenAI: Toward Effective Human-AI Collaboration
Sirui Han, Yuyao Zhang, Yidan Huang, Xueyan Li +2
2026-02-10
Artificial Intelligence · Computer Science
GenAI vs. Human Fact-Checkers: Accurate Ratings, Flawed Rationales
Yuehong Cassandra Tai, Khushi Navin Patni, Nicholas Daniel Hemauer, Bruce Desmarais +1
2025-02-27
Computers and Society · Computer Science
On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?
Kamil Malinka, Martin Perešíni, Anton Firc, Ondřej Hujňák +1
2023-07-04
Cryptography and Security · Computer Science
GPT in Sheep's Clothing: The Risk of Customized GPTs
Sagiv Antebi, Noam Azulay, Edan Habler, Ben Ganon +2
2024-01-18
Artificial Intelligence · Computer Science
First Proof
Mohammed Abouzaid, Andrew J. Blumberg, Martin Hairer, Joe Kileel +7
2026-03-17
Machine Learning · Computer Science
Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?
Albert Yu Sun, Eliott Zemour, Arushi Saxena, Udith Vaidyanathan +3
2024-04-17
Artificial Intelligence · Computer Science
Equitable Access to Justice: Logical LLMs Show Promise
Manuj Kant, Manav Kant, Marzieh Nabi, Preston Carlson +1
2024-10-15
Computers and Society · Computer Science
Unlocking the Black Box: Analysing the EU Artificial Intelligence Act's Framework for Explainability in AI
Georgios Pavlidis
2025-02-24