Response timing judgment is a critical component of interactive speech agents. Although there exists substantial prior work on turn modeling and voice wake-up, there is a lack of research on response timing judgments continuously aligned with user intent. To address this, we propose the Tap-to-Adapt framework, which enables users to naturally activate or interrupt the agent via tap interactions to construct online learning labels for response timing models. Under this framework, Dilated TCN and a sequential replay strategy play significant roles, as demonstrated through data-driven experiments and user studies. Additionally, we develop an evaluation and continuous data mining system tailored for the Tap-to-Adapt framework, through which we have collected approximately 20,000 samples from the user studies involving 20 participants.
@article{arxiv.2603.14449,
title = {Tap-to-Adapt: Learning User-Aligned Response Timing for Speech Agents},
author = {Zihong He and Hai-Ning Liang and Chen Liang},
journal= {arXiv preprint arXiv:2603.14449},
year = {2026}
}