Video dubbing aims to generate high-fidelity speech that is precisely temporally aligned with the visual content. Existing methods still suffer from limitations in speech naturalness and audio-visual synchronization, and are limited to monolingual settings. To address these challenges, we propose SyncVoice, a vision-augmented video dubbing framework built upon a pretrained text-to-speech (TTS) model. By fine-tuning the TTS model on audio-visual data, we achieve strong audiovisual consistency. We propose a Dual Speaker Encoder to effectively mitigate inter-language interference in cross-lingual speech synthesis and explore the application of video dubbing in video translation scenarios. Experimental results show that SyncVoice achieves high-fidelity speech generation with strong synchronization performance, demonstrating its potential in video dubbing tasks.
@article{arxiv.2512.05126,
title = {SyncVoice: Towards Video Dubbing with Vision-Augmented Pretrained TTS Model},
author = {Kaidi Wang and Yi He and Wenhao Guan and Weijie Wu and Hongwu Ding and Xiong Zhang and Di Wu and Meng Meng and Jian Luan and Lin Li and Qingyang Hong},
journal= {arXiv preprint arXiv:2512.05126},
year = {2025}
}