English

Direct Simultaneous Translation Activation for Large Audio-Language Models

Sound 2026-05-06 v2 Computation and Language Audio and Speech Processing

Abstract

Simultaneous speech-to-text translation (Simul-S2TT) aims to translate speech into target text in real time, outputting translations while receiving source speech input, rather than waiting for the entire utterance to be spoken. Simul-S2TT research often modifies model architectures to implement read-write strategies. However, with the rise of large audio-language models (LALMs), a key challenge is how to directly activate Simul-S2TT capabilities in base models without additional architectural changes. In this paper, we introduce {\bf Simul}taneous {\bf S}elf-{\bf A}ugmentation ({\bf SimulSA}), a strategy that utilizes LALMs' inherent capabilities to obtain simultaneous data by randomly truncating speech and constructing partially aligned translation. By incorporating them into offline SFT data, SimulSA effectively bridges the distribution gap between offline translation during pretraining and simultaneous translation during inference. Experimental results demonstrate that augmenting only about {\bf 1\%} of the simultaneous data, compared to the full offline SFT data, can significantly activate LALMs' Simul-S2TT capabilities without modifications to model architecture or decoding strategy.

Keywords

Cite

@article{arxiv.2509.15692,
  title  = {Direct Simultaneous Translation Activation for Large Audio-Language Models},
  author = {Pei Zhang and Yiming Wang and Jialong Tang and Baosong Yang and Rui Wang and Derek F. Wong and Fei Huang},
  journal= {arXiv preprint arXiv:2509.15692},
  year   = {2026}
}

Comments

Accepted by ICASSP 2026

R2 v1 2026-07-01T05:45:18.855Z