Conditional masked language model (CMLM) training has proven successful for non-autoregressive and semi-autoregressive sequence generation tasks, such as machine translation. Given a trained CMLM, however, it is not clear what the best inference strategy is. We formulate masked inference as a factorization of conditional probabilities of partial sequences, show that this does not harm performance, and investigate a number of simple heuristics motivated by this perspective. We identify a thresholding strategy that has advantages over the standard "mask-predict" algorithm, and provide analyses of its behavior on machine translation tasks.
@article{arxiv.2010.02352,
title = {Inference Strategies for Machine Translation with Conditional Masking},
author = {Julia Kreutzer and George Foster and Colin Cherry},
journal= {arXiv preprint arXiv:2010.02352},
year = {2020}
}