Fine-tuning pre-trained speech foundation models for Automatic Speech Recognition (ASR) is prevalent, yet constrained by substantial GPU memory requirements. We introduce ZO-ASR, a memory-efficient Zeroth-Order (ZO) method that avoids Back-Propagation (BP) and activation memory by estimating gradients via forward passes. When combined with SGD optimizer, ZO-ASR-SGD fine-tunes ASR models using only inference memory. Our evaluation spans supervised and unsupervised tasks. For Supervised Domain Adaptation on Whisper-Large-V3, ZO-ASR's multiple query mechanism enhances robustness and achieves up to an 18.9\% relative Word Error Rate reduction over zero-shot baselines, outperforming existing ZO methods. For unsupervised Test-Time Adaptation on Wav2Vec2-Base, ZO-ASR exhibits moderately lower performance compared to first-order optimizer Adam. Our BP-free approach provides a viable solution for fine-tuning ASR models in computationally resource-constrained or gradient-inaccessible scenarios.
@article{arxiv.2512.01267,
title = {ZO-ASR: Zeroth-Order Fine-Tuning of Speech Foundation Models without Back-Propagation},
author = {Yuezhang Peng and Yuxin Liu and Yao Li and Sheng Wang and Fei Wen and Xie Chen},
journal= {arXiv preprint arXiv:2512.01267},
year = {2025}
}
Comments
2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)