Beyond Video-to-SFX: Video to Audio Synthesis with Environmentally Aware Speech
Abstract
The generation of realistic, context-aware audio is important in real-world applications such as video game development. While existing video-to-audio (V2A) methods mainly focus on Foley sound generation, they struggle to produce intelligible speech. Meanwhile, current environmental speech synthesis approaches remain text-driven and fail to temporally align with dynamic video content. In this paper, we propose Beyond Video-to-SFX (BVS), a method to generate synchronized audio with environmentally aware intelligible speech for given videos. We introduce a two-stage modeling method: (1) stage one is a video-guided audio semantic (V2AS) model to predict unified audio semantic tokens conditioned on phonetic cues; (2) stage two is a video-conditioned semantic-to-acoustic (VS2A) model that refines semantic tokens into detailed acoustic tokens. Experiments demonstrate the effectiveness of BVS in scenarios such as video-to-context-aware speech synthesis and immersive audio background conversion, with ablation studies further validating our design. Our demonstration is available at~\href{https://xinleiniu.github.io/BVS-demo/}{BVS-Demo}.
Keywords
Cite
@article{arxiv.2509.15492,
title = {Beyond Video-to-SFX: Video to Audio Synthesis with Environmentally Aware Speech},
author = {Xinlei Niu and Jianbo Ma and Dylan Harper-Harris and Xiangyu Zhang and Charles Patrick Martin and Jing Zhang},
journal= {arXiv preprint arXiv:2509.15492},
year = {2025}
}