SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translation
Abstract
How to make human-interpreter-like read/write decisions for simultaneous speech translation (SimulST) systems? Current state-of-the-art systems formulate SimulST as a multi-turn dialogue task, requiring specialized interleaved training data and relying on computationally expensive large language model (LLM) inference for decision-making. In this paper, we propose SimulSense, a novel framework for SimulST that mimics human interpreters by continuously reading input speech and triggering write decisions to produce translation when a new sense unit is perceived. Experiments against two state-of-the-art baseline systems demonstrate that our proposed method achieves a superior quality-latency tradeoff and substantially improved real-time efficiency, where its decision-making is up to 9.6x faster than the baselines.
Cite
@article{arxiv.2509.21932,
title = {SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translation},
author = {Haotian Tan and Hiroki Ouchi and Sakriani Sakti},
journal= {arXiv preprint arXiv:2509.21932},
year = {2026}
}
Comments
\c{opyright} 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works