English

When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation

Software Engineering 2026-07-12 v1 Artificial Intelligence Machine Learning

Abstract

Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.

Cite

@article{arxiv.2607.10569,
  title  = {When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation},
  author = {Hong Yang and Qi Yu and Travis Desell},
  journal= {arXiv preprint arXiv:2607.10569},
  year   = {2026}
}

Comments

9 pages (excluding references), 4 figures, 4 tables. Accepted to the Agentic Software Engineering (SE 3.0) Workshop at KDD 2026 (non-archival)