Audio language models process audio inputs using textual prompts for tasks like speech recognition and audio captioning. Although built on multilingual pre-trained components, most are trained primarily on English, limiting their usability for other languages. This paper evaluates audio language models on Thai, a low-resource language, and finds that they lack emergent cross-lingual abilities despite their multilingual foundations. To address this, we explore data mixtures that optimize audio language models for both a target language and English while integrating audio comprehension and speech instruction-following into a unified model. Our experiments provide insights into improving instruction-following in low-resource languages by balancing language-specific and multilingual training data. The proposed model, Typhoon-Audio, significantly outperforms existing open-source models and achieves performance comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai.
@article{arxiv.2409.10999,
title = {Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models},
author = {Potsawee Manakul and Guangzhi Sun and Warit Sirichotedumrong and Kasima Tharnpipitchai and Kunat Pipatanakul},
journal= {arXiv preprint arXiv:2409.10999},
year = {2025}
}