Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation
Abstract
Representing manipulation actions as 2D trajectories in the camera plane provides a compact and interpretable basis for learning complex 3D manipulation policies. However, it also creates challenges from out-of-frame trajectories and limited precision. We propose Pix2Act, an imitation learning method that addresses these challenges by generating continuous image-space keypoint trajectories in each camera plane and losslessly recovering end-effector poses via triangulation. This reformulates high-dimensional 3D control as a simpler, more learnable 2D prediction problem. Crucially, it aligns observations and actions in the same coordinate space, enabling equivariant transformations to jointly rotate individual camera images together with their image-space actions. We analyze the symmetry properties of this augmentation and design a network architecture that can fuse multiple camera views while respecting their per-view rotations. As a result, Pix2Act implicitly enlarges the support of the data distribution and learns invariant action structures across transformations, yielding improved generalization and overall performance. Across diverse simulated and real-world manipulation tasks, Pix2Act outperforms state-of-the-art baselines and remains robust under camera perturbations.
Cite
@article{arxiv.2607.11167,
title = {Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation},
author = {Haojie Huang and Linfeng Zhao and Haotian Liu and Zhang Ye and Si-Yuan Huang and Mingxi Jia and Boce Hu and Fangzhou Lin and Yu Qi and Dian Wang and Robin Walters and Robert Platt},
journal= {arXiv preprint arXiv:2607.11167},
year = {2026}
}
Comments
Project Website: https://haojhuang.github.io/pix2act_page/