We introduce DiceHuBERT, a knowledge distillation framework for compressing HuBERT, a widely used self-supervised learning (SSL)-based speech foundation model. Unlike existing distillation methods that rely on layer-wise and feature-wise mapping between teacher and student models, DiceHuBERT leverages HuBERT's iterative self-distillation mechanism by directly replacing the original model with a student model. This replacement allows the student to be trained using the same SSL objective used when pre-training HuBERT, eliminating the need for additional modules or architectural constraints. Experimental results on SUPERB show that DiceHuBERT consistently outperforms existing distillation methods, improving phoneme recognition performance by over 21% and ASR performance by more than 14%. Furthermore, DiceHuBERT demonstrates competitive performance across multiple tasks, highlighting its clear advantage.
@article{arxiv.2507.02911,
title = {DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective},
author = {Hyung Gun Chi and Zakaria Aldeneh and Tatiana Likhomanenko and Oggi Rudovic and Takuya Higuchi and Li-Wei Chen and Shinji Watanabe and Ahmed Hussen Abdelaziz},
journal= {arXiv preprint arXiv:2507.02911},
year = {2025}
}