EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
Abstract
GUI agents must reason about how actions transform interface states, but end-to-end success rates entangle this ability with perception, grounding, planning, and recovery. We introduce EvoGUI, a diagnostic framework that converts normalized GUI trajectories into three complementary visual question answering probes: temporal ordering, inverse action/value prediction, and contrastive one-step successor discrimination. Their labels are derived from trajectory order and logged actions, requiring no additional task-label annotation after trajectory normalization. We instantiate EvoGUI-Bench from Mind2Web and WebLINX, yielding 3,000 instances across 120 domains, and evaluate 28 vision-language model configurations zero-shot. The strongest model reaches only 60.4 EvoGain, while model scale and GUI specialization do not reliably predict performance. These results establish EvoGUI-Bench as a scalable diagnostic complement to end-to-end GUI-agent evaluation while exposing substantial headroom in state-transition understanding. The source code is publicly available at https://github.com/Yyhhh6/EvoGUI.
Cite
@article{arxiv.2607.17050,
title = {EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding},
author = {Yaohan Yang and Minglei Shi and Borui Zhang and Jie Zhou and Jiwen Lu},
journal= {arXiv preprint arXiv:2607.17050},
year = {2026}
}