Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. However, the sequential text generation common pitfall with VAEs is that the model tends to ignore latent variables with a strong auto-regressive decoder. In this paper, we propose a principled approach to alleviate this issue by applying a discretized bottleneck to enforce an implicit latent feature matching in a more compact latent space. We impose a shared discrete latent space where each input is learned to choose a combination of latent atoms as a regularized latent representation. Our model endows a promising capability to model underlying semantics of discrete sequences and thus provide more interpretative latent structures. Empirically, we demonstrate our model's efficiency and effectiveness on a broad range of tasks, including language modeling, unaligned text style transfer, dialog response generation, and neural machine translation.
@article{arxiv.2004.10603,
title = {Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck},
author = {Yang Zhao and Ping Yu and Suchismit Mahapatra and Qinliang Su and Changyou Chen},
journal= {arXiv preprint arXiv:2004.10603},
year = {2021}
}