Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMath, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on HARDMath-mini, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like HARDMath to advance mathematical abilities of LLMs.
@article{arxiv.2410.09988,
title = {HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics},
author = {Jingxuan Fan and Sarah Martinson and Erik Y. Wang and Kaylie Hausknecht and Jonah Brenner and Danxian Liu and Nianli Peng and Corey Wang and Michael P. Brenner},
journal= {arXiv preprint arXiv:2410.09988},
year = {2024}
}
Comments
Code and the HARDMath dataset is available at https://github.com/sarahmart/HARDMath