English

RoboPlayground: Democratizing Robotic Evaluation through Structured Physical Domains

Robotics 2026-04-08 v1 Artificial Intelligence Computation and Language Human-Computer Interaction

Abstract

Evaluation of robotic manipulation systems has largely relied on fixed benchmarks authored by a small number of experts, where task instances, constraints, and success criteria are predefined and difficult to extend. This paradigm limits who can shape evaluation and obscures how policies respond to user-authored variations in task intent, constraints, and notions of success. We argue that evaluating modern manipulation policies requires reframing evaluation as a language-driven process over structured physical domains. We present RoboPlayground, a framework that enables users to author executable manipulation tasks using natural language within a structured physical domain. Natural language instructions are compiled into reproducible task specifications with explicit asset definitions, initialization distributions, and success predicates. Each instruction defines a structured family of related tasks, enabling controlled semantic and behavioral variation while preserving executability and comparability. We instantiate RoboPlayground in a structured block manipulation domain and evaluate it along three axes. A user study shows that the language-driven interface is easier to use and imposes lower cognitive workload than programming-based and code-assist baselines. Evaluating learned policies on language-defined task families reveals generalization failures that are not apparent under fixed benchmark evaluations. Finally, we show that task diversity scales with contributor diversity rather than task count alone, enabling evaluation spaces to grow continuously through crowd-authored contributions. Project Page: https://roboplayground.github.io

Keywords

Cite

@article{arxiv.2604.05226,
  title  = {RoboPlayground: Democratizing Robotic Evaluation through Structured Physical Domains},
  author = {Yi Ru Wang and Carter Ung and Evan Gubarev and Christopher Tan and Siddhartha Srinivasa and Dieter Fox},
  journal= {arXiv preprint arXiv:2604.05226},
  year   = {2026}
}

Comments

Yi Ru Wang and Carter Ung contributed equally

R2 v1 2026-07-01T11:56:17.293Z