Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applying high-frequency and multi-dimensional sensor data, such as eye-tracking data, leads to information loss and high token costs. To mitigate this, we investigate a visual prompting strategy that transforms sensor signals into data visualization images as an input to multimodal LLMs (MLLMs) using eye-tracking data. We conducted a systematic evaluation of MLLM-based HAR across three public eye-tracking datasets using three visualization types of timeline, heatmap, and scanpath, under varying temporal window sizes. Our findings suggest that visual prompting provides a token-efficient and scalable representation for eye-tracking data, highlighting its potential to enable MLLMs to effectively reason over high-frequency sensor signals in IoT contexts.
@article{arxiv.2604.09585,
title = {Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition},
author = {Jae Young Choi and Seon Gyeom Kim and Hyungjun Yoon and Taeckyung Lee and Donggun Lee and Jaeryung Chung and Jihyung Kil and Ryan Rossi and Sung-Ju Lee and Tak Yeon Lee},
journal= {arXiv preprint arXiv:2604.09585},
year = {2026}
}
Comments
6 pages. Conditionally accepted to IEEE PacificVis 2026 (VisNotes track)