Macro-block dropout for improved regularization in training end-to-end speech recognition models
Abstract
This paper proposes a new regularization algorithm referred to as macro-block dropout. The overfitting issue has been a difficult problem in training large neural network models. The dropout technique has proven to be simple yet very effective for regularization by preventing complex co-adaptations during training. In our work, we define a macro-block that contains a large number of units from the input to a Recurrent Neural Network (RNN). Rather than applying dropout to each unit, we apply random dropout to each macro-block. This algorithm has the effect of applying different drop out rates for each layer even if we keep a constant average dropout rate, which has better regularization effects. In our experiments using Recurrent Neural Network-Transducer (RNN-T), this algorithm shows relatively 4.30 % and 6.13 % Word Error Rates (WERs) improvement over the conventional dropout on LibriSpeech test-clean and test-other. With an Attention-based Encoder-Decoder (AED) model, this algorithm shows relatively 4.36 % and 5.85 % WERs improvement over the conventional dropout on the same test sets.
Cite
@article{arxiv.2212.14149,
title = {Macro-block dropout for improved regularization in training end-to-end speech recognition models},
author = {Chanwoo Kim and Sathish Indurti and Jinhwan Park and Wonyong Sung},
journal= {arXiv preprint arXiv:2212.14149},
year = {2023}
}
Comments
Accepted for presentation at The 2022 IEEE Spoken Language Technology Workshop (SLT 2022)