Recent advancements in large language models (LLMs) underscore the need for more comprehensive evaluation methods to accurately assess their reasoning capabilities. Existing benchmarks are often domain-specific and thus cannot fully capture an LLM's general reasoning potential. To address this limitation, we introduce the Knowledge Orthogonal Reasoning Gymnasium (KORGym), a dynamic evaluation platform inspired by KOR-Bench and Gymnasium. KORGym offers over fifty games in either textual or visual formats and supports interactive, multi-turn assessments with reinforcement learning scenarios. Using KORGym, we conduct extensive experiments on 19 LLMs and 8 VLMs, revealing consistent reasoning patterns within model families and demonstrating the superior performance of closed-source models. Further analysis examines the effects of modality, reasoning strategies, reinforcement learning techniques, and response length on model performance. We expect KORGym to become a valuable resource for advancing LLM reasoning research and developing evaluation methodologies suited to complex, interactive environments.
@article{arxiv.2505.14552,
title = {KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation},
author = {Jiajun Shi and Jian Yang and Jiaheng Liu and Xingyuan Bu and Jiangjie Chen and Junting Zhou and Kaijing Ma and Zhoufutu Wen and Bingli Wang and Yancheng He and Liang Song and Hualei Zhu and Shilong Li and Xingjian Wang and Wei Zhang and Ruibin Yuan and Yifan Yao and Wenjun Yang and Yunli Wang and Siyuan Fang and Siyu Yuan and Qianyu He and Xiangru Tang and Yingshui Tan and Wangchunshu Zhou and Zhaoxiang Zhang and Zhoujun Li and Wenhao Huang and Ge Zhang},
journal= {arXiv preprint arXiv:2505.14552},
year = {2025}
}