We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.
@article{arxiv.1902.04094,
title = {BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model},
author = {Alex Wang and Kyunghyun Cho},
journal= {arXiv preprint arXiv:1902.04094},
year = {2019}
}