Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.
@article{arxiv.2211.15089,
title = {Continuous diffusion for categorical data},
author = {Sander Dieleman and Laurent Sartran and Arman Roshannai and Nikolay Savinov and Yaroslav Ganin and Pierre H. Richemond and Arnaud Doucet and Robin Strudel and Chris Dyer and Conor Durkan and Curtis Hawthorne and Rémi Leblond and Will Grathwohl and Jonas Adler},
journal= {arXiv preprint arXiv:2211.15089},
year = {2022}
}
Comments
26 pages, 8 figures; corrections and additional information about hyperparameters