DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
摘要
Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.
引用
@article{arxiv.2605.23605,
title = {DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling},
author = {Jean-Marie Lemercier and Tomas Geffner and Karsten Kreis and Morteza Mardani and Arash Vahdat and Ante Jukić},
journal= {arXiv preprint arXiv:2605.23605},
year = {2026}
}