In this paper, we introduce a novel dIffusion language modEl pre-training framework for text generation, which we call GENIE. GENIE is a large-scale pretrained diffusion language model that consists of an encoder and a diffusion-based decoder, which can generate text by gradually transforming a random noise sequence into a coherent text sequence. To pre-train GENIE on a large-scale language corpus, we design a new continuous paragraph denoise objective, which encourages the diffusion-decoder to reconstruct a clean text paragraph from a corrupted version, while preserving the semantic and syntactic coherence. We evaluate GENIE on four downstream text generation benchmarks, namely XSum, CNN/DailyMail, Gigaword, and CommonGen. Our experimental results show that GENIE achieves comparable performance with the state-of-the-art autoregressive models on these benchmarks, and generates more diverse text samples. The code and models of GENIE are available at https://github.com/microsoft/ProphetNet/tree/master/GENIE.
@article{arxiv.2212.11685,
title = {Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise},
author = {Zhenghao Lin and Yeyun Gong and Yelong Shen and Tong Wu and Zhihao Fan and Chen Lin and Nan Duan and Weizhu Chen},
journal= {arXiv preprint arXiv:2212.11685},
year = {2023}
}
Comments
Previous version title -> GENIE: Large Scale Pre-training for Text Generation with Diffusion Model