Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
Abstract
We propose Kling-Foley, a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content. In Kling-Foley, we introduce multimodal diffusion transformers to model the interactions between video, audio, and text modalities, and combine it with a visual semantic representation module and an audio-visual synchronization module to enhance alignment capabilities. Specifically, these modules align video conditions with latent audio elements at the frame level, thereby improving semantic alignment and audio-visual synchronization. Together with text conditions, this integrated approach enables precise generation of video-matching sound effects. In addition, we propose a universal latent audio codec that can achieve high-quality modeling in various scenarios such as sound effects, speech, singing, and music. We employ a stereo rendering method that imbues synthesized audio with a spatial presence. At the same time, in order to make up for the incomplete types and annotations of the open-source benchmark, we also open-source an industrial-level benchmark Kling-Audio-Eval. Our experiments show that Kling-Foley trained with the flow matching objective achieves new audio-visual SOTA performance among public models in terms of distribution matching, semantic alignment, temporal alignment and audio quality.
Keywords
Cite
@article{arxiv.2506.19774,
title = {Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation},
author = {Jun Wang and Xijuan Zeng and Chunyu Qiang and Ruilong Chen and Shiyao Wang and Le Wang and Wangjing Zhou and Pengfei Cai and Jiahui Zhao and Nan Li and Zihan Li and Yuzhe Liang and Xiaopeng Wang and Haorui Zheng and Ming Wen and Kang Yin and Yiran Wang and Nan Li and Feng Deng and Liang Dong and Chen Zhang and Di Zhang and Kun Gai},
journal= {arXiv preprint arXiv:2506.19774},
year = {2025}
}