Conventional methods for aggregating layers in fine-tuned self-supervised speech models, such as using the final layer or weighted sum, suffer from information bottlenecks and static feature weighting for all dataset examples. We propose VARAN, a framework that dynamically tailors layer aggregation to individual inputs. By employing layer-specialized probing heads and data-dependent weighting, VARAN adaptively prioritizes layer's features based on input. Evaluations on automatic speech recognition and speech emotion recognition tasks demonstrate VARAN's superior performance, particularly when using the LoRA fine-tuning technique. The framework resolves the trade-off between preserving layer-specific information and enabling flexible feature utilization, advancing efficient adaptation of self-supervised speech representations.
@article{arxiv.2508.12061,
title = {VARAN: Variational Inference for Self-Supervised Speech Models Fine-Tuning on Downstream Tasks},
author = {Daria Diatlova and Nikita Balagansky and Alexander Varlamov and Egor Spirin},
journal= {arXiv preprint arXiv:2508.12061},
year = {2025}
}