The deep integration of large language models and automatic speech recognition systems has become a promising research direction with high practical value. To address the overfitting issue commonly observed in Low-Rank Adaptation (LoRA) during the supervised fine-tuning (SFT) stage, this work proposes an innovative training paradigm Iterative LoRA Training (ILT) in combination with an Iterative Pseudo Labeling strategy, effectively enhancing the theoretical upper bound of model performance. Based on Whisper-large-v3 and Qwen2-Audio, we conduct systematic experiments using a three-stage training process: Focus Training, Feed Back Training, and Fix Training. Experimental results demonstrate the effectiveness of the proposed method. Furthermore, the MegaAIS research team applied this technique in the Interspeech 2025 Multilingual Conversational Speech Language Modeling Challenge (MLC-SLM), achieving 4th in Track 1 (Multilingual ASR Task) and 1st place in Track 2 (Speech Separation and Recognition Task), showcasing the practical feasibility and strong application potential of our approach.
@article{arxiv.2507.08477,
title = {ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognition},
author = {Qingliang Meng and Hao Wu and Wei Liang and Wei Xu and Qing Zhao},
journal= {arXiv preprint arXiv:2507.08477},
year = {2025}
}
Comments
Accepted By Interspeech 2025 MLC-SLM workshop as a Research Paper