Temporal awareness is essential for video large language models (LLMs) to understand and reason about events within long videos, enabling applications like dense video captioning and temporal video grounding in a unified system. However, the scarcity of long videos with detailed captions and precise temporal annotations limits their temporal awareness. In this paper, we propose Seq2Time, a data-oriented training paradigm that leverages sequences of images and short video clips to enhance temporal awareness in long videos. By converting sequence positions into temporal annotations, we transform large-scale image and clip captioning datasets into sequences that mimic the temporal structure of long videos, enabling self-supervised training with abundant time-sensitive data. To enable sequence-to-time knowledge transfer, we introduce a novel time representation that unifies positional information across image sequences, clip sequences, and long videos. Experiments demonstrate the effectiveness of our method, achieving a 27.6% improvement in F1 score and 44.8% in CIDEr on the YouCook2 benchmark and a 14.7% increase in recall on the Charades-STA benchmark compared to the baseline.
@article{arxiv.2411.16932,
title = {Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding},
author = {Andong Deng and Zhongpai Gao and Anwesa Choudhuri and Benjamin Planche and Meng Zheng and Bin Wang and Terrence Chen and Chen Chen and Ziyan Wu},
journal= {arXiv preprint arXiv:2411.16932},
year = {2024}
}