SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
Abstract
What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed machine translation coverage beyond 200 languages, unified speech-to-speech translation models have yet to achieve similar strides. More specifically, conventional speech-to-speech translation systems rely on cascaded systems that perform translation progressively, putting high-performing unified systems out of reach. To address these gaps, we introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages. To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication
Cite
@article{arxiv.2308.11596,
title = {SeamlessM4T: Massively Multilingual & Multimodal Machine Translation},
author = {Seamless Communication and Loïc Barrault and Yu-An Chung and Mariano Cora Meglioli and David Dale and Ning Dong and Paul-Ambroise Duquenne and Hady Elsahar and Hongyu Gong and Kevin Heffernan and John Hoffman and Christopher Klaiber and Pengwei Li and Daniel Licht and Jean Maillard and Alice Rakotoarison and Kaushik Ram Sadagopan and Guillaume Wenzek and Ethan Ye and Bapi Akula and Peng-Jen Chen and Naji El Hachem and Brian Ellis and Gabriel Mejia Gonzalez and Justin Haaheim and Prangthip Hansanti and Russ Howes and Bernie Huang and Min-Jae Hwang and Hirofumi Inaguma and Somya Jain and Elahe Kalbassi and Amanda Kallet and Ilia Kulikov and Janice Lam and Daniel Li and Xutai Ma and Ruslan Mavlyutov and Benjamin Peloquin and Mohamed Ramadan and Abinesh Ramakrishnan and Anna Sun and Kevin Tran and Tuan Tran and Igor Tufanov and Vish Vogeti and Carleigh Wood and Yilin Yang and Bokai Yu and Pierre Andrews and Can Balioglu and Marta R. Costa-jussà and Onur Celebi and Maha Elbayad and Cynthia Gao and Francisco Guzmán and Justine Kao and Ann Lee and Alexandre Mourachko and Juan Pino and Sravya Popuri and Christophe Ropers and Safiyyah Saleem and Holger Schwenk and Paden Tomasello and Changhan Wang and Jeff Wang and Skyler Wang},
journal= {arXiv preprint arXiv:2308.11596},
year = {2023}
}