SpeechOp: Inference-Time Task Composition for Generative Speech Processing
Abstract
While generative Text-to-Speech (TTS) systems leverage vast ``in-the-wild" data to achieve remarkable success, speech-to-speech processing tasks like enhancement face data limitations, which lead data-hungry generative approaches to distort speech content and speaker identity. To bridge this gap, we present SpeechOp, a multi-task latent diffusion model that transforms pre-trained TTS models into a universal speech processor capable of performing a wide range of speech tasks and composing them in novel ways at inference time. By adapting a pre-trained TTS model, SpeechOp inherits a rich understanding of natural speech, accelerating training and improving S2S task quality, while simultaneously enhancing core TTS performance. Finally, we introduce Implicit Task Composition (ITC), a novel pipeline where ASR-derived transcripts (e.g., from Whisper) guide SpeechOp's enhancement via our principled inference-time task composition. ITC achieves state-of-the-art content preservation by robustly combining web-scale speech understanding with SpeechOp's generative capabilities. Audio samples are available at https://justinlovelace.github.io/projects/speechop
Cite
@article{arxiv.2509.14298,
title = {SpeechOp: Inference-Time Task Composition for Generative Speech Processing},
author = {Justin Lovelace and Rithesh Kumar and Jiaqi Su and Ke Chen and Kilian Q Weinberger and Zeyu Jin},
journal= {arXiv preprint arXiv:2509.14298},
year = {2025}
}