Large Language Model (LLM) agents are reshaping the game industry, by enabling more intelligent and human-preferable characters. Yet, current game benchmarks fall short of practical needs: they lack evaluations of diverse LLM capabilities across various game genres, studies of agentic modules crucial for complex gameplay, and fine-tuning datasets to adapt pre-trained LLMs into gaming agents. To fill these gaps, we present Orak, a benchmark for training and evaluating LLM agents across 12 popular video games spanning all major genres. Using a plug-and-play interface built on Model Context Protocol (MCP), Orak supports systematic and reproducible studies of agentic modules in varied game scenarios. We further release a fine-tuning dataset of expert LLM gameplay trajectories covering multiple genres, turning general LLMs into effective game agents. Orak offers a united evaluation framework, including game leaderboards, LLM battle arenas, and \fix{ablation studies} of input modality, agentic strategies, and fine-tuning effects, establishing a foundation towards versatile gaming agents. Code and datasets are available at https://github.com/krafton-ai/Orak and https://huggingface.co/datasets/KRAFTON/Orak.
@article{arxiv.2506.03610,
title = {Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games},
author = {Dongmin Park and Minkyu Kim and Beongjun Choi and Junhyuck Kim and Keon Lee and Jonghyun Lee and Inkyu Park and Byeong-Uk Lee and Jaeyoung Hwang and Jaewoo Ahn and Ameya S. Mahabaleshwarkar and Bilal Kartal and Pritam Biswas and Yoshi Suhara and Kangwook Lee and Jaewoong Cho},
journal= {arXiv preprint arXiv:2506.03610},
year = {2026}
}