Joint Source-Target Self Attention with Locality Constraints
Abstract
The dominant neural machine translation models are based on the encoder-decoder structure, and many of them rely on an unconstrained receptive field over source and target sequences. In this paper we study a new architecture that breaks with both conventions. Our simplified architecture consists in the decoder part of a transformer model, based on self-attention, but with locality constraints applied on the attention receptive field. As input for training, both source and target sentences are fed to the network, which is trained as a language model. At inference time, the target tokens are predicted autoregressively starting with the source sequence as previous tokens. The proposed model achieves a new state of the art of 35.7 BLEU on IWSLT'14 German-English and matches the best reported results in the literature on the WMT'14 English-German and WMT'14 English-French translation benchmarks.
Cite
@article{arxiv.1905.06596,
title = {Joint Source-Target Self Attention with Locality Constraints},
author = {José A. R. Fonollosa and Noe Casas and Marta R. Costa-jussà},
journal= {arXiv preprint arXiv:1905.06596},
year = {2019}
}