We explore the application of very deep Transformer models for Neural Machine Translation (NMT). Using a simple yet effective initialization technique that stabilizes training, we show that it is feasible to build standard Transformer-based models with up to 60 encoder layers and 12 decoder layers. These deep models outperform their baseline 6-layer counterparts by as much as 2.5 BLEU, and achieve new state-of-the-art benchmark results on WMT14 English-French (43.8 BLEU and 46.4 BLEU with back-translation) and WMT14 English-German (30.1 BLEU).The code and trained models will be publicly available at: https://github.com/namisan/exdeep-nmt.
@article{arxiv.2008.07772,
title = {Very Deep Transformers for Neural Machine Translation},
author = {Xiaodong Liu and Kevin Duh and Liyuan Liu and Jianfeng Gao},
journal= {arXiv preprint arXiv:2008.07772},
year = {2020}
}
Comments
6 pages, 3 figures and 4 tables. V2 includes the back-translation results