Deep encoders have been proven to be effective in improving neural machine translation (NMT) systems, but it reaches the upper bound of translation quality when the number of encoder layers exceeds 18. Worse still, deeper networks consume a lot of memory, making it impossible to train efficiently. In this paper, we present Symbiosis Networks, which include a full network as the Symbiosis Main Network (M-Net) and another shared sub-network with the same structure but less layers as the Symbiotic Sub Network (S-Net). We adopt Symbiosis Networks on Transformer-deep (m-n) architecture and define a particular regularization loss Lτ between the M-Net and S-Net in NMT. We apply joint-training on the Symbiosis Networks and aim to improve the M-Net performance. Our proposed training strategy improves Transformer-deep (12-6) by 0.61, 0.49 and 0.69 BLEU over the baselines under classic training on WMT'14 EN->DE, DE->EN and EN->FR tasks. Furthermore, our Transformer-deep (12-6) even outperforms classic Transformer-deep (18-6).
@article{arxiv.2112.11642,
title = {Joint-training on Symbiosis Networks for Deep Nueral Machine Translation models},
author = {Zhengzhe Yu and Jiaxin Guo and Minghan Wang and Daimeng Wei and Hengchao Shang and Zongyao Li and Zhanglin Wu and Yuxia Wang and Yimeng Chen and Chang Su and Min Zhang and Lizhi Lei and shimin tao and Hao Yang},
journal= {arXiv preprint arXiv:2112.11642},
year = {2021}
}