English

Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment

Machine Learning 2026-05-18 v2 Artificial Intelligence Human-Computer Interaction

Abstract

Test-Time Scaling (TTS), which samples multiple candidate actions and ranks them via a Critic Model, has emerged as a promising paradigm for generalist GUI agents. Its efficacy thus hinges on the critic's fine-grained ranking ability. However, existing GUI critic models uniformly adopt binary classification. Our motivational analysis of these models exposes a severe entanglement: scores for valid actions and plausible-but-invalid distractors become indistinguishable. We attribute this failure to two structural defects: Affordance Collapse--the hierarchical affordance space is compressed into 0/1 labels; and Noise Sensitivity--binary objectives overfit to noisy decision boundaries. To resolve this, we introduce BBCritic (Beyond-Binary Critic), a paradigm shift grounded in the Functional Equivalence Hypothesis. Through two-stage contrastive learning, BBCritic aligns instructions and actions in a shared Affordance Space, recovering the hierarchical structure that binary supervision flattens. We also present BBBench (Beyond-Binary Bench), the first GUI critic benchmark that pairs a dense action space with a hierarchical four-level taxonomy, enabling fine-grained ranking evaluation. Experimental results show that BBCritic-3B, trained without any extra annotation, outperforms 7B-parameter SOTA binary models. It demonstrates strong zero-shot transferability across platforms and tasks, supporting our methodological view: GUI critique is fundamentally a metric-learning problem, not a classification one.

Keywords

Cite

@article{arxiv.2605.14311,
  title  = {Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment},
  author = {Yuchen Sun and Pei Fu and Shaojie Zhang and Anan Du and Xiuwen Xi and Ruoceng Zhang and Zhenbo Luo and Jian Luan and Chongyang Zhang},
  journal= {arXiv preprint arXiv:2605.14311},
  year   = {2026}
}

Comments

28 pages including appendix. Code and BBBench benchmark to be released