Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs
Abstract
In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.
Keywords
Cite
@article{arxiv.2506.08633,
title = {Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs},
author = {Šimon Sedláček and Bolaji Yusuf and Ján Švec and Pradyoth Hegde and Santosh Kesiraju and Oldřich Plchot and Jan Černocký},
journal= {arXiv preprint arXiv:2506.08633},
year = {2025}
}
Comments
Accepted to Interspeech 2025