LAE-ST-MoE：用于 E2E 双语混合语 ASR 的提升语言感知编码器
摘要
最近, 为缓解 code-switching (CS) 自动语音识别 (ASR) 中不同语言之间的混淆, 条件因子化模型, 如语言感知编码器 (LAE), 明确地忽略不同语言之间的情境信息。However, this information may be helpful for ASR modeling。To alleviate this issue, we propose the LAE-ST-MoE framework。It incorporates speech translation (ST) tasks into LAE and utilizes ST to learn the contextual information between different languages。It introduces a task-based mixture of expert modules, employing separate feed-forward networks for the ASR and ST tasks。Experimental results on the ASRU 2019 Mandarin-English CS challenge dataset demonstrate that, compared to the LAE-based CTC, the LAE-ST-MoE model achieves a 9.26% mix error reduction on the CS test with the same decoding parameter。Moreover, the well-trained LAE-ST-MoE model can perform ST tasks from CS speech to Mandarin or English text。
引用
@article{arxiv.2309.16178,
title = {LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR},
author = {Guodong Ma and Wenxuan Wang and Yuke Li and Yuting Yang and Binbin Du and Haoran Fu},
journal= {arXiv preprint arXiv:2309.16178},
year = {2023}
}
备注
Accepted to IEEE ASRU 2023