SeamlessM4T：大规模多语言与多模态机器翻译
计算与语言
2023-10-26 v3
摘要
要创造能帮人们在任意两种语言间翻译语音的“巴别鱼”需要什么？尽管近期基于文本的模型突破已将机器翻译覆盖推至200种语言以上，统一的语音到语音翻译模型尚未取得类似进展。更具体地说，传统语音到语音翻译系统依赖级联系统逐步执行翻译，使高性能统一系统难以企及。为弥补这些差距，我们提出SeamlessM4T，一个支持多达100种语言的语音到语音翻译、语音到文本翻译、文本到语音翻译、文本到文本翻译以及自动语音识别的单一模型。为此，我们使用100万小时开放语音音频数据通过w2v-BERT 2.0学习自监督语音表示。随后，我们创建了自动对齐语音翻译的多模态语料库。经筛选并与人工标注及伪标注数据结合，我们开发了首个能进行英语与语音和文本双向翻译的多语言系统。在FLEURS上，SeamlessM4T为翻译至多种目标语言树立新标准，在直接语音到文本翻译上较先前SOTA提升20% BLEU。相较强级联模型，SeamlessM4T在语音到文本中将英译质量提升1.3 BLEU点，在语音到语音中提升2.6 ASR-BLEU点。经鲁棒性测试，在语音到文本任务中，相较当前SOTA模型，我们的系统对背景噪声与说话人变化表现更优。关键的是，我们评估了SeamlessM4T的性别偏见与附加毒性以评估翻译安全性。最后，本工作所有贡献均已开源，可访问 https://github.com/facebookresearch/seamless_communication
引用
@article{arxiv.2308.11596,
title = {SeamlessM4T: Massively Multilingual & Multimodal Machine Translation},
author = {Seamless Communication and Loïc Barrault and Yu-An Chung and Mariano Cora Meglioli and David Dale and Ning Dong and Paul-Ambroise Duquenne and Hady Elsahar and Hongyu Gong and Kevin Heffernan and John Hoffman and Christopher Klaiber and Pengwei Li and Daniel Licht and Jean Maillard and Alice Rakotoarison and Kaushik Ram Sadagopan and Guillaume Wenzek and Ethan Ye and Bapi Akula and Peng-Jen Chen and Naji El Hachem and Brian Ellis and Gabriel Mejia Gonzalez and Justin Haaheim and Prangthip Hansanti and Russ Howes and Bernie Huang and Min-Jae Hwang and Hirofumi Inaguma and Somya Jain and Elahe Kalbassi and Amanda Kallet and Ilia Kulikov and Janice Lam and Daniel Li and Xutai Ma and Ruslan Mavlyutov and Benjamin Peloquin and Mohamed Ramadan and Abinesh Ramakrishnan and Anna Sun and Kevin Tran and Tuan Tran and Igor Tufanov and Vish Vogeti and Carleigh Wood and Yilin Yang and Bokai Yu and Pierre Andrews and Can Balioglu and Marta R. Costa-jussà and Onur Celebi and Maha Elbayad and Cynthia Gao and Francisco Guzmán and Justine Kao and Ann Lee and Alexandre Mourachko and Juan Pino and Sravya Popuri and Christophe Ropers and Safiyyah Saleem and Holger Schwenk and Paden Tomasello and Changhan Wang and Jeff Wang and Skyler Wang},
journal= {arXiv preprint arXiv:2308.11596},
year = {2023}
}