Evaluating LLMs Code Reasoning Under Real-World Context

Changshu Liu

Evaluating LLMs Code Reasoning Under Real-World Context

Software Engineering 2026-04-15 v1

Authors: Changshu Liu

Abstract

Code reasoning tasks are increasingly crucial to evaluating large language models (LLMs). Yet most existing benchmarks rely on simplistic, LLM-generated snippets or human-written solutions to code challenges and often restrict inputs and outputs to primitive types, failing to reflect the structure and dependencies of real-world projects. These simplifications limit their ability to measure practical generalizability. We present R2Eval1, a benchmark of 135 code reasoning problems drawn from ten widely used Python projects. Unlike prior work, R2Eval serializes compound and custom types, preserving real-world data complexity and enabling a more realistic assessment of LLMs.

Keywords

code generation large language model evaluation large language model reasoning

Cite

@article{arxiv.2604.12881,
  title  = {Evaluating LLMs Code Reasoning Under Real-World Context},
  author = {Changshu Liu},
  journal= {arXiv preprint arXiv:2604.12881},
  year   = {2026}
}

Comments

Accepted by ICES SRC (ACM Student Research Competition)

Related papers

View all related →