English

ST-$\pi$: Structured SpatioTemporal VLA for Robotic Manipulation

Robotics 2026-04-21 v1 Computer Vision and Pattern Recognition

Abstract

Vision-language-action (VLA) models have achieved great success on general robotic tasks, but still face challenges in fine-grained spatiotemporal manipulation. Typically, existing methods mainly embed spatiotemporal knowledge into visual and action representations, and directly perform a cross-modal mapping for step-level action prediction. However, such spatiotemporal reasoning remains largely implicit, making it difficult to handle multiple sequential behaviors with explicit spatiotemporal boundaries. In this work, we propose ST-π\pi, a structured spatiotemporal VLA model for robotic manipulation. Our model is guided by two key designs: 1) Spatiotemporal VLM. We encode 4D observations and task instructions into latent spaces, and feed them into the LLM to generate a sequence of causally ordered chunk-level action prompts consisting of sub-tasks, spatial grounding and temporal grounding. 2) Spatiotemporal action expert. Conditioned on chunk-level action prompts, we design a structured dual-generator guidance to jointly model spatial dependencies and temporal causality, thus predicting step-level action parameters. Within this structured framework, the VLM explicitly plans global spatiotemporal behavior, and the action expert further refines local spatiotemporal control. In addition, we propose a real-world robotic dataset with structured spatiotemporal annotations for fine-tuning. Extensive experiments have been conducted to demonstrate the effectiveness of our model. Our code link: https://github.com/chuanhaoma/ST-pi.

Keywords

Cite

@article{arxiv.2604.17880,
  title  = {ST-$\pi$: Structured SpatioTemporal VLA for Robotic Manipulation},
  author = {Chuanhao Ma and Hanyu Zhou and Shihan Peng and Yan Li and Tao Gu and Luxin Yan},
  journal= {arXiv preprint arXiv:2604.17880},
  year   = {2026}
}
R2 v1 2026-07-01T12:17:44.803Z