English

VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents

Computation and Language 2026-04-06 v3

Abstract

With the rapid progress of multimodal large language models, operating system (OS) agents become increasingly capable of automating tasks through on-device graphical user interfaces (GUIs). However, most existing OS agents are designed for idealized settings, whereas real-world environments often present untrustworthy conditions. To mitigate risks of over-execution in such scenarios, we propose a query-driven human-agent-GUI interaction framework that enables OS agents to decide when to query humans for more reliable task completion. Built upon this framework, we introduce VeriOS-Agent, a trustworthy OS agent trained with a three-stage learning paradigm that falicitate the decoupling and utilization of meta-knowledge by supervised fine-tuning and group relative policy optimization. Concretely, VeriOS-Agent autonomously executes actions in normal conditions while proactively querying humans in untrustworthy scenarios. Experiments show that VeriOS-Agent improves the average step-wise success rate by 19.72\% in over the strongest baselines, without compromising normal performance. VeriOS-Agent significantly improves performance in untrustworthy scenarios while maintaining comparable performance in trustworthy scenarios. Analysis highlights VeriOS-Agent's rationality, generalizability, and scalability. The codes, datasets and models are available at https://github.com/Wuzheng02/VeriOS.

Keywords

Cite

@article{arxiv.2509.07553,
  title  = {VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents},
  author = {Zheng Wu and Heyuan Huang and Xingyu Lou and Xiangmou Qu and Pengzhou Cheng and Zongru Wu and Weiwen Liu and Weinan Zhang and Jun Wang and Zhaoxiang Wang and Zhuosheng Zhang},
  journal= {arXiv preprint arXiv:2509.07553},
  year   = {2026}
}
R2 v1 2026-07-01T05:28:05.129Z