English

Mouse-Guided Gaze: Semi-Supervised Learning of Intention-Aware Representations for Reading Detection

Human-Computer Interaction 2025-09-25 v1

Abstract

Understanding user intent during magnified reading is critical for accessible interface design. Yet magnification collapses visual context and forces continual viewport dragging, producing fragmented, noisy gaze and obscuring reading intent. We present a semi-supervised framework that learns intention-aware gaze representations by leveraging mouse trajectories as weak supervision. The model is first pretrained to predict mouse velocity from unlabeled gaze, then fine-tuned to classify reading versus scanning. To address magnification-induced distortions, we jointly model raw gaze within the magnified viewport and a compensated view remapped to the original screen, which restores spatial continuity across lines and paragraphs. Across text and webpage datasets, our approach consistently outperforms supervised baselines, with semi-supervised pretraining yielding up to 7.5% F1 improvement in challenging settings. These findings highlight the value of behavior-driven pretraining for robust, gaze-only interaction, paving the way for adaptive, hands-free accessibility tools.

Keywords

Cite

@article{arxiv.2509.19574,
  title  = {Mouse-Guided Gaze: Semi-Supervised Learning of Intention-Aware Representations for Reading Detection},
  author = {Seongsil Heo and Roberto Manduchi},
  journal= {arXiv preprint arXiv:2509.19574},
  year   = {2025}
}

Comments

Accepted at NeurIPS 2025 Foundation Models for the Brain and Body NeurIPS 2025 Workshop

R2 v1 2026-07-01T05:53:09.440Z