Evaluating the reasoning abilities of large language models (LLMs) solely from final answers can obscure failures in intermediate steps, especially in multi-hop QA benchmarks without step-level annotations. To address this gap, we introduce Omanic, an open-domain 4-hop QA benchmark designed not only to measure final-answer accuracy but also to diagnose where reasoning breaks down. Omanic contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench), with each evaluation question decomposed into single-hop sub-questions, intermediate answers, and structured graph topologies. Experiments with proprietary and open-source LLMs show that Omanic is challenging, while step-wise analysis reveals a later-hop bottleneck, factual knowledge floor, and error propagation along reasoning chains. Fine-tuning on OmanicSynth transfers to six reasoning and mathematics benchmarks, yielding a 7.41-point average gain and validating its effectiveness as supervision for reasoning-capability transfer. We release the data at https://huggingface.co/datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.
@article{arxiv.2603.16654,
title = {Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models},
author = {Xiaojie Gu and Sherry T. Tong and Aosong Feng and Sophia Simeng Han and Jinghui Lu and Yingjian Chen and Yusuke Iwasawa and Yutaka Matsuo and Chanjun Park and Rex Ying and Irene Li},
journal= {arXiv preprint arXiv:2603.16654},
year = {2026}
}