English

SecAgent: Efficient Mobile GUI Agent with Semantic Context

Computer Vision and Pattern Recognition 2026-04-01 v2

Abstract

Mobile Graphical User Interface (GUI) agents powered by multimodal large language models have demonstrated promising capabilities in automating complex smartphone tasks. However, existing approaches face two critical limitations: the scarcity of high-quality multilingual datasets, particularly for non-English ecosystems, and inefficient history representation methods. To address these challenges, we present SecAgent, an efficient mobile GUI agent at 3B scale. We first construct a human-verified Chinese mobile GUI dataset with 18k grounding samples and 121k navigation steps across 44 applications, along with a Chinese navigation benchmark featuring multi-choice action annotations. Building upon this dataset, we propose a semantic context mechanism that distills history screenshots and actions into concise, natural language summaries, significantly reducing computational costs while preserving task-relevant information. Through supervised and reinforcement fine-tuning, SecAgent outperforms similar-scale baselines and achieves performance comparable to 7B-8B models on our and public navigation benchmarks. Our dataset is available at https://huggingface.co/datasets/alibabagroup/CMGUI.

Keywords

Cite

@article{arxiv.2603.08533,
  title  = {SecAgent: Efficient Mobile GUI Agent with Semantic Context},
  author = {Yiping Xie and Song Chen and Jingxuan Xing and Wei Jiang and Zekun Zhu and Yingyao Wang and Pi Bu and Jun Song and Yuning Jiang and Bo Zheng},
  journal= {arXiv preprint arXiv:2603.08533},
  year   = {2026}
}
R2 v1 2026-07-01T11:10:34.167Z