English
Related papers

Related papers: XBOUND: Exploring Capability Boundaries of Device-…

200 papers

To realize the full potential of quantum technologies, finding good strategies to control quantum information processing devices in real time becomes increasingly important. Usually these strategies require a precise understanding of the…

Agentic AI systems plan, use tools, maintain state, and act across multi-step workflows with external effects, meaning trustworthy deployment can no longer be judged by task completion alone. The current literature remains fragmented across…

Software Engineering · Computer Science 2026-04-23 Christopher Koch , Joshua Andreas Wellbrock

The paradigm of agentic AI is shifting from engineered complex workflows to post-training native models. However, existing agents are typically confined to static, predefined action spaces--such as exclusively using APIs, GUI events, or…

Machine Learning · Computer Science 2025-12-11 Kaichen He , Zihao Wang , Muyao Li , Anji Liu , Yitao Liang

Autonomous agents capable of navigating Graphical User Interfaces (GUIs) hold the potential to revolutionize digital productivity. However, achieving true digital autonomy extends beyond reactive element matching; it necessitates a…

Computer Vision and Pattern Recognition · Computer Science 2026-04-28 Hongxin Li , Xiping Wang , Jingran Su , Zheng Ju , Yuntao Chen , Qing Li , Zhaoxiang Zhang

With AI agents increasingly deployed as long-running systems, it becomes essential to autonomously construct and continuously evolve customized software to enable interaction within dynamic environments. Yet, existing benchmarks evaluate…

Computer-using agents (CUAs) enable task completion through natural interaction with operating systems and software interfaces. While script-based verifiers are widely adopted for evaluation, they suffer from limited scalability and…

Software Engineering · Computer Science 2025-10-22 Haojia Lin , Xiaoyu Tan , Yulei Qin , Zihan Xu , Yuchen Shi , Zongyi Li , Gang Li , Shaofei Cai , Siqi Cai , Chaoyou Fu , Ke Li , Xing Sun

Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations, but this hybrid action space often leaves them uncertain about when to continue with…

Artificial Intelligence · Computer Science 2026-05-13 Xuhao Hu , Xi Zhang , Haiyang Xu , Kyle Qiao , Jingyi Yang , Xuanjing Huang , Jing Shao , Ming Yan , Jieping Ye

GUI agent aims to enable automated operations on Mobile/PC devices, which is an important task toward achieving artificial general intelligence. The rapid advancement of VLMs accelerates the development of GUI agents, owing to their…

Computer Vision and Pattern Recognition · Computer Science 2025-09-01 Zhixiong Zeng , Jing Huang , Liming Zheng , Wenkang Han , Yufeng Zhong , Lei Chen , Longrong Yang , Yingjie Chu , Yuzhi He , Lin Ma

AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation…

Artificial Intelligence · Computer Science 2026-02-24 Stephan Rabanser , Sayash Kapoor , Peter Kirgis , Kangheng Liu , Saiteja Utpala , Arvind Narayanan

Agentic systems are becoming more capable: agents define strategies, take actions, and interact with different environments. This autonomy poses serious challenges for overseeing and assessing agent behavior. Most current tools are limited,…

Computation and Language · Computer Science 2026-05-22 Asaf Yehudai , Lilach Eden , Michal Shmueli-Scheuer

AI agents are increasingly used to solve complex, multi-step tasks, but existing multi-agent frameworks remain brittle as workflows grow in scale and depth. Small errors at intermediate stages can propagate through agent interactions, while…

Artificial Intelligence · Computer Science 2026-05-26 Andy Xu , Yu-Wing Tai

The realization of Artificial General Intelligence (AGI) necessitates Embodied AI agents capable of robust spatial perception, effective task planning, and adaptive execution in physical environments. However, current large language models…

As autonomous agents become adept at understanding and interacting with graphical user interface (GUI) environments, a new era of automated task execution is emerging. Recent studies have demonstrated that Reinforcement Learning (RL) can…

Artificial Intelligence · Computer Science 2026-03-16 Songqin Nong , Xiaoxuan Tang , Jingxuan Xu , Sheng Zhou , Jianfeng Chen , Tao Jiang , Wenhao Xu

Automation systems that can autonomously drive application user interfaces to complete user tasks are of great benefit, especially when users are situationally or permanently impaired. Prior automation systems do not produce generalizable…

Human-Computer Interaction · Computer Science 2024-07-01 Wei Li , Fu-Lin Hsu , Will Bishop , Folawiyo Campbell-Ajala , Max Lin , Oriana Riva

Recent advances in deep reinforcement learning and scalable photorealistic simulation have led to increasingly mature embodied AI for various visual tasks, including navigation. However, while impressive progress has been made for teaching…

Robotics · Computer Science 2023-10-16 Naoki Yokoyama , Qian Luo , Dhruv Batra , Sehoon Ha

We present a framework that lets a service provider achieve end-to-end management objectives under varying load. Dynamic control actions are performed by a reinforcement learning (RL) agent. Our work includes experimentation and evaluation…

Machine Learning · Computer Science 2022-10-11 Forough Shahab Samani , Rolf Stadler

Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby boosting human productivity. Existing works leveraging Large Language Model (LLM) or LLM-based AI agents have shown…

Computer Vision and Pattern Recognition · Computer Science 2024-01-02 Difei Gao , Lei Ji , Zechen Bai , Mingyu Ouyang , Peiran Li , Dongxing Mao , Qinchen Wu , Weichen Zhang , Peiyi Wang , Xiangwu Guo , Hengxu Wang , Luowei Zhou , Mike Zheng Shou

General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measured how agent architecture shapes performance across heterogeneous protocols and diverse…

The evaluation of Vision-Language-Action (VLA) agents is hindered by the coarse, end-task success metric that fails to provide precise skill diagnosis or measure robustness to real-world perturbations. This challenge is exacerbated by a…

Robotics · Computer Science 2025-10-22 Jierui Peng , Yanyan Zhang , Yicheng Duan , Tuo Liang , Vipin Chaudhary , Yu Yin

Deep reinforcement learning has been applied successfully to solve various real-world problems and the number of its applications in the multi-agent settings has been increasing. Multi-agent learning distinctly poses significant challenges…

Machine Learning · Computer Science 2021-02-24 Ngoc Duy Nguyen , Thanh Thi Nguyen , Doug Creighton , Saeid Nahavandi