MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation

Yifan Peng; Ilia Kulikov; Yilin Yang; Sravya Popuri; Hui Lu; Changhan Wang; Hongyu Gong

MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation

Computation and Language 2024-03-20 v1 Sound Audio and Speech Processing

Authors: Yifan Peng , Ilia Kulikov , Yilin Yang , Sravya Popuri , Hui Lu , Changhan Wang , Hongyu Gong

Abstract

There have been emerging research interest and advances in speech-to-speech translation (S2ST), translating utterances from one language to another. This work proposes Multitask Speech Language Model (MSLM), which is a decoder-only speech language model trained in a multitask setting. Without reliance on text training data, our model is able to support multilingual S2ST with speaker style preserved.

Keywords

speech-to-speech translation multitask speech language model decoder-only speaker style preservation textless multilingual

Cite

@article{arxiv.2403.12408,
  title  = {MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation},
  author = {Yifan Peng and Ilia Kulikov and Yilin Yang and Sravya Popuri and Hui Lu and Changhan Wang and Hongyu Gong},
  journal= {arXiv preprint arXiv:2403.12408},
  year   = {2024}
}

MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation

Abstract

Keywords

Cite

Related papers