Speech dysfluency modeling is a task to detect dysfluencies in speech, such as repetition, block, insertion, replacement, and deletion. Most recent advancements treat this problem as a time-based object detection problem. In this work, we revisit this problem from a new perspective: tokenizing dysfluencies and modeling the detection problem as a token-based automatic speech recognition (ASR) problem. We propose rule-based speech and text dysfluency simulators and develop VCTK-token, and then develop a Whisper-like seq2seq architecture to build a new benchmark with decent performance. We also systematically compare our proposed token-based methods with time-based methods, and propose a unified benchmark to facilitate future research endeavors. We open-source these resources for the broader scientific community. The project page is available at https://rorizzz.github.io/
@article{arxiv.2409.13582,
title = {Time and Tokens: Benchmarking End-to-End Speech Dysfluency Detection},
author = {Xuanru Zhou and Jiachen Lian and Cheol Jun Cho and Jingwen Liu and Zongli Ye and Jinming Zhang and Brittany Morin and David Baquirin and Jet Vonk and Zoe Ezzes and Zachary Miller and Maria Luisa Gorno Tempini and Gopala Anumanchipalli},
journal= {arXiv preprint arXiv:2409.13582},
year = {2024}
}