Testing GPT-4 with Wolfram Alpha and Code Interpreter plug-ins on math and science problems
Abstract
This report describes a test of the large language model GPT-4 with the Wolfram Alpha and the Code Interpreter plug-ins on 105 original problems in science and math, at the high school and college levels, carried out in June-August 2023. Our tests suggest that the plug-ins significantly enhance GPT's ability to solve these problems. Having said that, there are still often "interface" failures; that is, GPT often has trouble formulating problems in a way that elicits useful answers from the plug-ins. Fixing these interface failures seems like a central challenge in making GPT a reliable tool for college-level calculation problems.
Keywords
Cite
@article{arxiv.2308.05713,
title = {Testing GPT-4 with Wolfram Alpha and Code Interpreter plug-ins on math and science problems},
author = {Ernest Davis and Scott Aaronson},
journal= {arXiv preprint arXiv:2308.05713},
year = {2025}
}
Comments
Update September 2024: This revised version corrects earlier minor errors in the Arbitrary Numerical problems 3 and 17. See pages 18 and 24. Update Feb. 2025: This revised version updates the reference to (Ye et al. 2023)