English

Vidformer: Drop-in Declarative Optimization for Rendering Video-Native Query Results

Databases 2026-01-27 v1 Multimedia

Abstract

When interactively exploring video data, video-native querying involves consuming query results as videos, including steps such as compilation of extracted video clips or data overlays. These video-native queries are bottlenecked by rendering, not the execution of the underlying queries. This rendering is currently performed using post-processing scripts that are often slow. This step poses a critical point of friction in interactive video data workloads: even short clips contain thousands of high-definition frames; conventional OpenCV/Python scripts must decode -> transform -> encode the entire data stream before a single pixel appears, leaving users waiting for many seconds, minutes, or hours. To address these issues, we present Vidformer, a drop-in rendering accelerator for video-native querying which, (i) transparently lifts existing visualization code into a declarative representation, (ii) transparently optimizes and parallelizes rendering, and (iii) instantly serves videos through a Video on Demand protocol with just-in-time segment rendering. We demonstrate that Vidformer cuts full-render time by 2-3x across diverse annotation workloads, and, more critically, drops time-to-playback to 0.25-0.5s. This represents a 400x improvement that decouples clip length from first-frame playback latency, and unlocks the ability to perform interactive video-native querying with sub-second latencies. Furthermore, we show how our approach enables interactive video-native LLM-based conversational querying as well.

Keywords

Cite

@article{arxiv.2601.17221,
  title  = {Vidformer: Drop-in Declarative Optimization for Rendering Video-Native Query Results},
  author = {Dominik Winecki and Arnab Nandi},
  journal= {arXiv preprint arXiv:2601.17221},
  year   = {2026}
}
R2 v1 2026-07-01T09:18:08.283Z