ORCESTRA: VLM-driven Visual Robot programming in Mixed Reality

Ivan Snegirev; Elizaveta Semenyakina; Mikhail Konenkov; Artem Lykov; Miguel Altamirano Cabrera; Dzmitry Tsetserukou

ORCESTRA: VLM-driven Visual Robot programming in Mixed Reality

Robotics 2026-08-01 v1 Computer Vision and Pattern Recognition Human-Computer Interaction

Authors: Ivan Snegirev , Elizaveta Semenyakina , Mikhail Konenkov , Artem Lykov , Miguel Altamirano Cabrera , Dzmitry Tsetserukou

View on arXiv ↗ PDF ↗

Abstract

ORCESTRA is a mixed-reality system for programming robot digital twins through no-code waypoint teaching and language-guided control. In a passthrough mixed-reality workspace, users place robot twins on real surfaces, teach trajectories, save robot-relative episodes, or issue spoken/typed commands that a vision-language model converts into structured digital-twin plans. Both interaction modes share a backend for metric grounding, embodiment-aware validation, preview, confirmation, and digital-twin execution. The system supports heterogeneous robot embodiments, including fixed-base manipulators, a mobile base, and a humanoid robot, demonstrating MR validation as a safety layer for language-guided robot programming before physical deployment.

Cite

@article{arxiv.2608.00775,
  title  = {ORCESTRA: VLM-driven Visual Robot programming in Mixed Reality},
  author = {Ivan Snegirev and Elizaveta Semenyakina and Mikhail Konenkov and Artem Lykov and Miguel Altamirano Cabrera and Dzmitry Tsetserukou},
  journal= {arXiv preprint arXiv:2608.00775},
  year   = {2026}
}

Comments

4 page, 3 figures, 1 table, ISMAR 2026