English

GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation

Artificial Intelligence 2025-11-03 v1 Computer Vision and Pattern Recognition

Abstract

While Multimodal Large Language Models (MLLMs) have advanced GUI navigation agents, current approaches face limitations in cross-domain generalization and effective history utilization. We present a reasoning-enhanced framework that systematically integrates structured reasoning, action prediction, and history summarization. The structured reasoning component generates coherent Chain-of-Thought analyses combining progress estimation and decision reasoning, which inform both immediate action predictions and compact history summaries for future steps. Based on this framework, we train a GUI agent, \textbf{GUI-Rise}, through supervised fine-tuning on pseudo-labeled trajectories and reinforcement learning with Group Relative Policy Optimization (GRPO). This framework employs specialized rewards, including a history-aware objective, directly linking summary quality to subsequent action performance. Comprehensive evaluations on standard benchmarks demonstrate state-of-the-art results under identical training data conditions, with particularly strong performance in out-of-domain scenarios. These findings validate our framework's ability to maintain robust reasoning and generalization across diverse GUI navigation tasks. Code is available at https://leon022.github.io/GUI-Rise.

Keywords

Cite

@article{arxiv.2510.27210,
  title  = {GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation},
  author = {Tao Liu and Chongyu Wang and Rongjie Li and Yingchen Yu and Xuming He and Bai Song},
  journal= {arXiv preprint arXiv:2510.27210},
  year   = {2025}
}

Comments

Published in NeurIPS 2025

R2 v1 2026-07-01T07:15:09.968Z