Interactive assistance systems typically provide feedback after an action has been completed, supporting error recovery but not preventing the error itself. We present TRAFA, a real-time predictive feedback system for procedural tasks that intervenes before errors are committed. TRAFA operationalizes predictive feedback through a Track-Forecast-Act framework that tracks hand and object state, forecasts user motion conditioned on scene context, and triggers feedback when a predicted action is likely to violate task constraints. We instantiate this pipeline in a sequential assembly setting and evaluate it through both technical benchmarking and a controlled user study against conventional reactive feedback. Our results show that predictive feedback improves task accuracy and efficiency while maintaining a comparable number of feedback events. These findings position feedback timing as a key dimension in system design and show how real-time anticipation can be integrated into interactive systems to prevent errors before they occur.
@article{arxiv.2605.24526,
title = {TRAFA: Anticipating User Actions to Reduce Errors in Procedural Tasks with Predictive Feedback},
author = {Sassan Mokhtar and Lars Doorenbos and Fatemeh Jabbari and Marius Bock and Dominik Bach and Juergen Gall},
journal= {arXiv preprint arXiv:2605.24526},
year = {2026}
}