This paper introduces a highly efficient greedy decoding algorithm for Transducer-based speech recognition models. We redesign the standard nested-loop design for RNN-T decoding, swapping loops over frames and labels: the outer loop iterates over labels, while the inner loop iterates over frames searching for the next non-blank symbol. Additionally, we represent partial hypotheses in a special structure using CUDA tensors, supporting parallelized hypotheses manipulations. Experiments show that the label-looping algorithm is up to 2.0X faster than conventional batched decoding when using batch size 32. It can be further combined with other compiler or GPU call-related techniques to achieve even more speedup. Our algorithm is general-purpose and can work with both conventional Transducers and Token-and-Duration Transducers. We open-source our implementation to benefit the research community.
@article{arxiv.2406.06220,
title = {Label-Looping: Highly Efficient Decoding for Transducers},
author = {Vladimir Bataev and Hainan Xu and Daniel Galvez and Vitaly Lavrukhin and Boris Ginsburg},
journal= {arXiv preprint arXiv:2406.06220},
year = {2025}
}