ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solving
Abstract
To enhance large language models (LLMs) for chemistry problem solving, several LLM-based agents augmented with tools have been proposed, such as ChemCrow and Coscientist. However, their evaluations are narrow in scope, leaving a large gap in understanding the benefits of tools across diverse chemistry tasks. To bridge this gap, we develop ChemToolAgent, an enhanced chemistry agent over ChemCrow, and conduct a comprehensive evaluation of its performance on both specialized chemistry tasks and general chemistry questions. Surprisingly, ChemToolAgent does not consistently outperform its base LLMs without tools. Our error analysis with a chemistry expert suggests that: For specialized chemistry tasks, such as synthesis prediction, we should augment agents with specialized tools; however, for general chemistry questions like those in exams, agents' ability to reason correctly with chemistry knowledge matters more, and tool augmentation does not always help.
Keywords
Cite
@article{arxiv.2411.07228,
title = {ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solving},
author = {Botao Yu and Frazier N. Baker and Ziru Chen and Garrett Herb and Boyu Gou and Daniel Adu-Ampratwum and Xia Ning and Huan Sun},
journal= {arXiv preprint arXiv:2411.07228},
year = {2025}
}
Comments
Accepted to NAACL 2025 Findings. Previous title: Tooling or Not Tooling? The Impact of Tools on Language Agents for Chemistry Problem Solving. Based on the camera ready version, this version adds more experimental results. We also release the toolkit in ChemMCP (https://osu-nlp-group.github.io/ChemMCP), which is a continuously updated and MCP-compatible chemistry toolkit