Scene understanding is central to general physical intelligence, and video is a primary modality for capturing both state and temporal dynamics of a scene. Yet understanding physical processes remains difficult, as models must combine object localization, hand-object interactions, relational parsing, temporal reasoning, and step-level procedural inference. Existing benchmarks usually evaluate these capabilities separately, limiting diagnosis of why models fail on procedural tasks. We introduce BARISTA, a densely annotated egocentric dataset and benchmark of 185 real-world coffee-preparation videos covering fully automatic, portafilter-based, and capsule-based workflows. BARISTA provides verified per-frame scene graphs linking persistent object identities to masks, tracks, boxes, attributes, typed relations, hand-object interactions, activities, and process steps. From these graphs, we derive zero-shot language-based tasks spanning phrase grounding, hand-object interaction recognition, referring, activity recognition, relation extraction, and temporal visual question answering. Experiments reveal strong variation across task families and no consistently dominant model family, positioning BARISTA as a challenging diagnostic benchmark for procedural video understanding. Code and dataset available at https://huggingface.co/datasets/ramblr/BARISTA.
@article{arxiv.2605.12074,
title = {BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding},
author = {Patrick Knab and Orgest Xhelili and Inis Buzi and Drago Andres Guggiana Nilo and Mohd Saquib Khan and Lorenz Kolb and Manuel Scherzer and Kerem Yildirir and Christian Bartelt and Philipp Johannes Schubert},
journal= {arXiv preprint arXiv:2605.12074},
year = {2026}
}