Static In, Dynamic Out: Counterfactual Action Augmentation for Moving Object Manipulation
Abstract
Visuomotor policies have advanced on manipulation tasks where the target object stays static during execution, but real deployments break this assumption: parts drift on conveyors and fruits sway in the wind. We introduce Static In, Dynamic Out (SIDO), a counterfactual action augmentation that enables a policy trained only on static object demonstrations to adapt to unseen object motion at test time. Our key idea is to factorize moving object manipulation into two sub-problems: predicting where the object will be, and reaching that predicted pose. SIDO displaces the object to a counterfactual future position and morphs the demonstrated action chunk to preserve the hand-object relative pose, yielding a goal-conditioned policy. At deployment an object pose predictor supplies the future position. Across three simulated tasks (Mug, Square, Stack) under five object motion patterns and two real-world tasks (Gantry, Peachtree), SIDO improves moving object success over the baselines while preserving static object performance. Project website: https://sido-staticindynamicout.github.io/.
Cite
@article{arxiv.2607.27890,
title = {Static In, Dynamic Out: Counterfactual Action Augmentation for Moving Object Manipulation},
author = {Woo Chul Shin and Zhenyang Chen and Alfred Cueva and Nadun Ranawaka Arachchige and Yingyan Celine Lin and Benjamin Joffe and Shreyas Kousik and Danfei Xu},
journal= {arXiv preprint arXiv:2607.27890},
year = {2026}
}