Deepfake Word Detection by Next-token Prediction using Fine-tuned Whisper
Abstract
Deepfake speech utterances can be forged by replacing one or more words in a bona fide utterance with semantically different words synthesized with speech-generative models. While a dedicated synthetic word detector could be developed, we developed a cost-effective method that fine-tunes a pre-trained Whisper model to detect synthetic words while transcribing the input utterance via next-token prediction. We further investigate using partially vocoded utterances as the fine-tuning data, thus reducing the cost of data collection. Our experiments demonstrate that, on in-domain test data, the fine-tuned Whisper yields low synthetic-word detection error rates and transcription error rates. On out-of-domain test data with synthetic words produced with unseen speech-generative models, the fine-tuned Whisper remains on par with a dedicated ResNet-based detection model; however, the overall performance degradation calls for strategies to improve its generalization capability.
Keywords
Cite
@article{arxiv.2602.22658,
title = {Deepfake Word Detection by Next-token Prediction using Fine-tuned Whisper},
author = {Hoan My Tran and Xin Wang and Wanying Ge and Xuechen Liu and Junichi Yamagishi},
journal= {arXiv preprint arXiv:2602.22658},
year = {2026}
}
Comments
Submitted to Interspeech. To quote: Interspeech no longer enforces an anonymity period for submissions. While uploading a version online is permitted, your official submission to Interspeech must not contain any author-identifying information. ... a note indicating that the paper was submitted for review to (or, eventually, accepted at) Interspeech should be included in the posting