"Code-as-Policy" considers how executable code can complement data-intensive Vision-Language-Action (VLA) methods, yet their effectiveness as autonomous controllers for embodied manipulation remains underexplored. We present CaP-X, an open-access framework for systematically studying Code-as-Policy agents in robot manipulation. At its core is CaP-Gym, an interactive environment in which agents control robots by synthesizing and executing programs that compose perception and control primitives. Building on this foundation, CaP-Bench evaluates frontier language and vision-language models across varying levels of abstraction, interaction, and perceptual grounding. Across 12 models, CaP-Bench reveals a consistent trend: performance improves with human-crafted abstractions but degrades as these priors are removed, exposing a dependence on designer scaffolding. At the same time, we observe that this gap can be mitigated through scaling agentic test-time computation--through multi-turn interaction, structured execution feedback, visual differencing, automatic skill synthesis, and ensembled reasoning--substantially improves robustness even when agents operate over low-level primitives. These findings allow us to derive CaP-Agent0, a training-free framework that recovers human-level reliability on several manipulation tasks in simulation and on real embodiments. We further introduce CaP-RL, showing reinforcement learning with verifiable rewards improves success rates and transfers from sim2real with minimal gap. Together, CaP-X provides a principled, open-access platform for advancing embodied coding agents.
@article{arxiv.2603.22435,
title = {CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation},
author = {Max Fu and Justin Yu and Karim El-Refai and Ethan Kou and Haoru Xue and Huang Huang and Wenli Xiao and Guanzhi Wang and Fei-Fei Li and Guanya Shi and Jiajun Wu and Shankar Sastry and Yuke Zhu and Ken Goldberg and Linxi "Jim" Fan},
journal= {arXiv preprint arXiv:2603.22435},
year = {2026}
}