A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations
Abstract
Speech Large Language Models (Speech LLMs) have emerged as a crucial paradigm in recent years, extending the capabilities of traditional LLMs to speech tasks such as automatic speech recognition (ASR) and spoken dialogue modeling. However, their effectiveness in real-world multilingual conversations remains limited by the scarcity of data that captures natural conversational phenomena. To address this, the MLC-SLM Challenge provides a multilingual conversational dataset and evaluates models on two tasks: ASR with oracle segmentation (Task I) and joint diarization and recognition without oracle information (Task II). In this paper, we focus on Task II and propose a unified speech LLM that jointly performs diarization and ASR in an end-to-end manner. By reformulating the training data format and modifying the inference procedure, our model addresses the ambiguity inherent in pre-segmented audio and achieves a 54.87\% relative improvement in tcpWER/tcpCER over the baseline, ranking 8th overall, despite using a smaller LLM backbone. We also report results from Task I using a fine-tuned speech LLM.
Cite
@article{arxiv.2507.02927,
title = {A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations},
author = {Phurich Saengthong and Boonnithi Jiaramaneepinit and Sheng Li and Manabu Okumura and Takahiro Shinozaki},
journal= {arXiv preprint arXiv:2507.02927},
year = {2025}
}