T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS
Abstract
Neural audio codecs provide promising acoustic features for speech synthesis, with representative streaming codecs like Mimi providing high-quality acoustic features for real-time Text-to-Speech (TTS) applications. However, Mimi's decoder, which employs a hybrid transformer and convolution architecture, introduces significant latency bottlenecks on edge devices due to the the compute intensive nature of deconvolution layers which are not friendly for mobile-CPUs, such as the most representative framework XNNPACK. This paper introduces T-Mimi, a novel modification of the Mimi codec decoder that replaces its convolutional components with a purely transformer-based decoder, inspired by the TS3-Codec architecture. This change dramatically reduces on-device TTS latency from 42.1ms to just 4.4ms. Furthermore, we conduct quantization aware training and derive a crucial finding: the final two transformer layers and the concluding linear layers of the decoder, which are close to the waveform, are highly sensitive to quantization and must be preserved at full precision to maintain audio quality.
Cite
@article{arxiv.2601.20094,
title = {T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS},
author = {Haibin Wu and Bach Viet Do and Naveen Suda and Julian Chan and Madhavan C R and Gene-Ping Yang and Yi-Chiao Wu and Naoyuki Kanda and Yossef Adi and Xin Lei and Yue Liu and Florian Metze and Yuzong Liu},
journal= {arXiv preprint arXiv:2601.20094},
year = {2026}
}
Comments
Accepted by ICASSP 2026