In August 2023, Scott Aaronson and I reported the results of testing GPT4 with the Wolfram Alpha and Code Interpreter plug-ins over a collection of 105 original high-school level and college-level science and math problems (Davis and Aaronson, 2023). In September 2024, I tested the recently released model GPT-4o1-preview on the same collection. Overall I found that performance had significantly improved, but was still considerably short of perfect. In particular, problems that involve spatial reasoning are often stumbling blocks.
@article{arxiv.2410.22340,
title = {Testing GPT-4-o1-preview on math and science problems: A follow-up study},
author = {Ernest Davis},
journal= {arXiv preprint arXiv:2410.22340},
year = {2024}
}