Self-supervised learning (SSL) for speech representation has been successfully applied in various downstream tasks, such as speech and speaker recognition. More recently, speech SSL models have also been shown to be beneficial in advancing spoken language understanding tasks, implying that the SSL models have the potential to learn not only acoustic but also linguistic information. In this paper, we aim to clarify if speech SSL techniques can well capture linguistic knowledge. For this purpose, we introduce SpeechGLUE, a speech version of the General Language Understanding Evaluation (GLUE) benchmark. Since GLUE comprises a variety of natural language understanding tasks, SpeechGLUE can elucidate the degree of linguistic ability of speech SSL models. Experiments demonstrate that speech SSL models, although inferior to text-based SSL models, perform better than baselines, suggesting that they can acquire a certain amount of general linguistic knowledge from just unlabeled speech data.
@article{arxiv.2306.08374,
title = {SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?},
author = {Takanori Ashihara and Takafumi Moriya and Kohei Matsuura and Tomohiro Tanaka and Yusuke Ijima and Taichi Asami and Marc Delcroix and Yukinori Honma},
journal= {arXiv preprint arXiv:2306.08374},
year = {2024}
}
Comments
Accepted at INTERSPEECH 2023. This paper has been extended in a subsequent journal paper, see https://ieeexplore.ieee.org/abstract/document/10597571