English

AdaLTM: Adaptive Layer-wise Task Vector Merging for Categorical Speech Emotion Recognition with ASR Knowledge Integration

Audio and Speech Processing 2026-03-27 v1

Abstract

Integrating Automatic Speech Recognition (ASR) into Speech Emotion Recognition (SER) enhances modeling by providing linguistic context. However, conventional feature fusion faces performance bottlenecks, and multi-task learning often suffers from optimization conflicts. While task vectors and model merging have addressed such conflicts in NLP and CV, their potential in speech tasks remains largely unexplored. In this work, we propose an Adaptive Layer-wise Task Vector Merging (AdaLTM) framework based on WavLM-Large. Instead of joint optimization, we extract task vectors from in-domain ASR and SER models fine-tuned on emotion datasets. These vectors are integrated into a frozen base model using layer-wise learnable coefficients. This strategy enables depth-aware balancing of linguistic and paralinguistic knowledge across transformer layers without gradient interference. Experiments on the MSP-Podcast demonstrate that the proposed approach effectively mitigates conflicts between ASR and SER.

Keywords

Cite

@article{arxiv.2603.25041,
  title  = {AdaLTM: Adaptive Layer-wise Task Vector Merging for Categorical Speech Emotion Recognition with ASR Knowledge Integration},
  author = {Chia-Yu Lee and Huang-Cheng Chou and Tzu-Quan Lin and Yuanchao Li and Ya-Tse Wu and Shrikanth Narayanan and Chi-Chun Lee},
  journal= {arXiv preprint arXiv:2603.25041},
  year   = {2026}
}

Comments

Submitted to Interspeech 2026

R2 v1 2026-07-01T11:38:32.814Z