English

Towards Visually-Guided Movie Subtitle Translation for Indic Languages

Computation and Language 2026-05-13 v1

Abstract

Movie subtitle translation is inherently multimodal, yet text-only systems often miss visual cues needed to convey emotion, action, and social nuance, especially for low-resource Indic languages (English to Hindi, Bengali, Telugu, Tamil and Kannada). We present a case study on five full-length films and compare two lightweight visual grounding strategies: structured attribute summaries from a 5-minute sliding window and free-text summaries of inter-subtitle visual gaps. Our analysis shows that temporal misalignment between subtitles and frames is a major obstacle in long-form video, often rendering indiscriminate visual grounding ineffective. However, oracle selective grounding, which replaces only the lowest-quality 20-30\% of baseline segments with visual-enhanced outputs, consistently improves COMET over the text-only baseline while requiring far less visual processing. Among the two approaches, coarse attribute-based visual context summarization is more robust, capturing scene-level emotion and contextual subtle cues that text alone often misses

Keywords

Cite

@article{arxiv.2605.11993,
  title  = {Towards Visually-Guided Movie Subtitle Translation for Indic Languages},
  author = {Tarun Chintada and Kshetrimayum Boynao Singh and Asif Ekbal},
  journal= {arXiv preprint arXiv:2605.11993},
  year   = {2026}
}