We introduce KALL-E, a novel autoregressive (AR) language model for text-to-speech (TTS) synthesis that operates by predicting the next distribution of continuous speech frames. Unlike existing methods, KALL-E directly models the continuous speech distribution conditioned on text, eliminating the need for any diffusion-based components. Specifically, we utilize a Flow-VAE to extract a continuous latent speech representation from waveforms, instead of relying on discrete speech tokens. A single AR Transformer is then trained to predict these continuous speech distributions from text, optimizing a Kullback-Leibler divergence loss as its objective. Experimental results demonstrate that KALL-E achieves superior speech synthesis quality and can even adapt to a target speaker from just a single sample. Importantly, KALL-E provides a more direct and effective approach for utilizing continuous speech representations in TTS.
@article{arxiv.2412.16846,
title = {KALL-E:Autoregressive Speech Synthesis with Next-Distribution Prediction},
author = {Kangxiang Xia and Xinfa Zhu and Jixun Yao and Wenjie Tian and Wenhao Li and Lei Xie},
journal= {arXiv preprint arXiv:2412.16846},
year = {2025}
}