Simultaneous speech-to-text translation systems must balance translation quality with latency. Although quality evaluation is well established, latency measurement remains a challenge. Existing metrics produce inconsistent results, especially in short-form settings with artificial presegmentation. We present the first comprehensive meta-evaluation of latency metrics across language pairs and systems. We uncover a structural bias in current metrics related to segmentation. We introduce YAAL (Yet Another Average Lagging) for a more accurate short-form evaluation and LongYAAL for unsegmented audio. We propose SoftSegmenter, a resegmentation tool based on soft word-level alignment. We show that YAAL and LongYAAL, together with SoftSegmenter, outperform popular latency metrics, enabling more reliable assessments of short- and long-form simultaneous speech translation systems. We implement all artifacts within the OmniSTEval toolkit: https://github.com/pe-trik/OmniSTEval.
@article{arxiv.2509.17349,
title = {Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation},
author = {Peter Polák and Sara Papi and Luisa Bentivogli and Ondřej Bojar},
journal= {arXiv preprint arXiv:2509.17349},
year = {2026}
}
Comments
Changes: - small change in the name (Evaluation -> Meta-Evaluation); - added reference to the implementation; - excluded two test sets (IWSLT22 En-Zh, En-Ja) because of incorrect and missing segmentation; - main results unchanged; - added Degenerate Policy Test; - added sensitivity of the metrics to change in the metric value