English

FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

Computation and Language 2026-01-21 v1 Computer Vision and Pattern Recognition Multimedia

Abstract

Although Multimodal Large Language Models (MLLMs) demonstrate strong omni-modal perception, their ability to forecast future events from audio-visual cues remains largely unexplored, as existing benchmarks focus mainly on retrospective understanding. To bridge this gap, we introduce FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments. The evaluated models are required to perform cross-modal causal and temporal reasoning, as well as effectively leverage internal knowledge to predict future events. FutureOmni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains. Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios, with the best accuracy of 64.8% achieved by Gemini 3 Flash. To mitigate this limitation, we curate a 7K-sample instruction-tuning dataset and propose an Omni-Modal Future Forecasting (OFF) training strategy. Evaluations on FutureOmni and popular audio-visual and video-only benchmarks demonstrate that OFF enhances future forecasting and generalization. We publicly release all code (https://github.com/OpenMOSS/FutureOmni) and datasets (https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni).

Keywords

Cite

@article{arxiv.2601.13836,
  title  = {FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs},
  author = {Qian Chen and Jinlan Fu and Changsong Li and See-Kiong Ng and Xipeng Qiu},
  journal= {arXiv preprint arXiv:2601.13836},
  year   = {2026}
}

Comments

https://openmoss.github.io/FutureOmni

R2 v1 2026-07-01T09:12:16.512Z