We present ORCA (Omni Research on Calculation in AI) Benchmark - a novel benchmark that evaluates large language models (LLMs) on multi-domain, real-life quantitative reasoning using verified outputs from Omni's calculator engine. In 500 natural-language tasks across domains such as finance, physics, health, and statistics, the five state-of-the-art systems (ChatGPT-5, Gemini~2.5~Flash, Claude~Sonnet~4.5, Grok~4, and DeepSeek~V3.2) achieved only 45–63% accuracy, with errors mainly related to rounding (35%) and calculation mistakes (33%). Results in specific domains indicate strengths in mathematics and engineering, but weaknesses in physics and natural sciences. Correlation analysis (r≈0.40–0.65) shows that the models often fail together but differ in the types of errors they make, highlighting their partial complementarity rather than redundancy. Unlike standard math datasets, ORCA evaluates step-by-step reasoning, numerical precision, and domain generalization across real problems from finance, physics, health, and statistics.
@article{arxiv.2511.02589,
title = {The ORCA Benchmark: Evaluating Real-World Calculation Accuracy in Large Language Models},
author = {Claudia Herambourg and Dawid Siuda and Julia Kopczyńska and Joao R. L. Santos and Wojciech Sas and Joanna Śmietańska-Nowak},
journal= {arXiv preprint arXiv:2511.02589},
year = {2025}
}