FunAudioLLM：面向人类与LLM自然交互的语音理解与生成基础模型
声音
2024-07-12 v3 人工智能
音频与语音处理
摘要
本报告介绍FunAudioLLM，这是一个旨在增强人类与大语言模型（LLM）之间自然语音交互的模型系列。其核心包括两个创新模型：SenseVoice，负责多语言语音识别、情感识别和音频事件检测；CosyVoice，实现自然语音生成，可控制多语言、语音色彩、说话风格和说话人身份。SenseVoice-Small提供了5种语言的超低延迟语音识别，SenseVoice-Large支持超过50种语言的高精度语音识别，而CosyVoice在多语言语音生成、零样本情境学习、跨语言语音克隆和指令遵循能力方面表现出色。与SenseVoice和CosyVoice相关的模型已在Modelscope和Huggingface上开源，相应的训练、推理和微调代码也已在GitHub上发布。通过将这些模型与LLM集成，FunAudioLLM实现了语音到语音翻译、情感语音聊天、交互式播客和富有表现力的有声书朗读等应用，推动了语音交互技术的边界。演示可访问https://fun-audio-llm.github.io，代码可访问https://github.com/FunAudioLLM。
引用
@article{arxiv.2407.04051,
title = {FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs},
author = {Keyu An and Qian Chen and Chong Deng and Zhihao Du and Changfeng Gao and Zhifu Gao and Yue Gu and Ting He and Hangrui Hu and Kai Hu and Shengpeng Ji and Yabin Li and Zerui Li and Heng Lu and Haoneng Luo and Xiang Lv and Bin Ma and Ziyang Ma and Chongjia Ni and Changhe Song and Jiaqi Shi and Xian Shi and Hao Wang and Wen Wang and Yuxuan Wang and Zhangyu Xiao and Zhijie Yan and Yexin Yang and Bin Zhang and Qinglin Zhang and Shiliang Zhang and Nan Zhao and Siqi Zheng},
journal= {arXiv preprint arXiv:2407.04051},
year = {2024}
}
备注
Work in progress. Authors are listed in alphabetical order by family name