English
Related papers

Related papers: OS-MAP: How Far Can Computer-Using Agents Go in Br…

200 papers

Large language models are shifting from passive information providers to active agents intended for complex workflows. However, their deployment as reliable AI workers in enterprise is stalled by benchmarks that fail to capture the…

Multimodal Large Language Models (MLLMs) are evolving from passive observers into active agents, solving problems through Visual Expansion (invoking visual tools) and Knowledge Expansion (open-web search). However, existing evaluations fall…

Artificial Intelligence · Computer Science 2026-04-06 Qianshan Wei , Yishan Yang , Siyi Wang , Jinglin Chen , Binyu Wang , Jiaming Wang , Shuang Chen , Zechen Li , Yang Shi , Yuqi Tang , Weining Wang , Yi Yu , Chaoyou Fu , Qi Li , Yi-Fan Zhang

Multimodal agents are making rapid progress on general computer-use tasks, yet existing benchmarks remain largely confined to browsers and basic desktop applications, falling short in professional software workflows that dominate real-world…

Software Engineering · Computer Science 2026-01-07 Jiaxin Ai , Yukang Feng , Fanrui Zhang , Jianwen Sun , Zizhen Li , Chuanhao Li , Yifan Chang , Wenxiao Wu , Ruoxi Wang , Mingliang Zhai , Kaipeng Zhang

AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the…

With VLM-powered computer-using agents (CUAs) becoming increasingly capable at graphical user interface (GUI) navigation and manipulation, reliable step-level decision-making has emerged as a key bottleneck for real-world deployment. In…

Artificial Intelligence · Computer Science 2025-12-19 Zhenyu Wu , Jingjing Xie , Zehao Li , Bowen Yang , Qiushi Sun , Zhaoyang Liu , Zhoumianze Liu , Yu Qiao , Xiangyu Yue , Zun Wang , Zichen Ding

In this paper, we introduce OSUniverse: a benchmark of complex, multimodal desktop-oriented tasks for advanced GUI-navigation AI agents that focuses on ease of use, extensibility, comprehensive coverage of test cases, and automated…

Artificial Intelligence · Computer Science 2025-05-07 Mariya Davydova , Daniel Jeffries , Patrick Barker , Arturo Márquez Flores , Sinéad Ryan

LLM-powered agents are both a promising new technology and a source of complexity, where choices about models, tools, and prompting can affect their usefulness. While numerous benchmarks measure agent accuracy across domains, they mostly…

The rapid development in the field of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents to assist humans in their daily tasks. However, a significant gap remains in assessing…

Computation and Language · Computer Science 2024-02-26 Negar Arabzadeh , Julia Kiseleva , Qingyun Wu , Chi Wang , Ahmed Awadallah , Victor Dibia , Adam Fourney , Charles Clarke

We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an…

Machine Learning · Computer Science 2026-02-24 Ayush Nangia , Shikhar Mishra , Aman Gokrani , Paras Chopra

The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks…

Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. However, existing benchmarks remain focused on single agentic capability, failing to capture…

Artificial Intelligence · Computer Science 2026-04-24 Keyu Li , Junhao Shi , Yang Xiao , Mohan Jiang , Jie Sun , Yunze Wu , Dayuan Fu , Shijie Xia , Xiaojie Cai , Tianze Xu , Weiye Si , Wenjie Li , Dequan Wang , Pengfei Liu

Unlike traditional automation tools or static LLM-based systems, agents combine decision-making and tool utilization to accomplish complex tasks, showing great potential in software engineering. However, existing studies largely focus on…

Software Engineering · Computer Science 2025-11-04 Zhuowen Yin , Cuifeng Gao , Chunsong Fan , Wenzhang Yang , Yinxing Xue , Lijun Zhang

While agentic AI has advanced in automating individual tasks, managing complex multi-agent workflows remains a challenging problem. This paper presents a research vision for autonomous agentic systems that orchestrate collaboration within…

Artificial Intelligence · Computer Science 2025-10-06 Charlie Masters , Advaith Vellanki , Jiangbo Shangguan , Bart Kultys , Jonathan Gilmore , Alastair Moore , Stefano V. Albrecht

The rapid deployment of Large language model (LLM) agents in critical domains like healthcare and finance necessitates robust security frameworks. To address the absence of standardized evaluation benchmarks for these agents in dynamic…

Cryptography and Security · Computer Science 2025-06-19 Yuchuan Fu , Xiaohan Yuan , Dongxia Wang

The comprehension and adoption of Artificial Intelligence (AI) are beset with practical and ethical problems. This article presents a 5-level AI Capability Assessment Model (AI-CAM) and a related AI Capabilities Matrix (AI-CM) to assist…

Computers and Society · Computer Science 2023-05-26 Butler , Tom , Espinoza-Limón , Angelina , Seppälä , Selja

LLM-based agents struggle to execute complex, multi-step Standard Operating Procedures (SOPs) that are fundamental to industrial automation. Existing benchmarks fail to capture the procedural complexity and tool orchestration demands of…

Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby boosting human productivity. Existing works leveraging Large Language Model (LLM) or LLM-based AI agents have shown…

Computer Vision and Pattern Recognition · Computer Science 2024-01-02 Difei Gao , Lei Ji , Zechen Bai , Mingyu Ouyang , Peiran Li , Dongxing Mao , Qinchen Wu , Weichen Zhang , Peiyi Wang , Xiangwu Guo , Hengxu Wang , Luowei Zhou , Mike Zheng Shou

Recent advances in large language models (LLMs) have enabled the emergence of general-purpose agents for automating end-to-end machine learning (ML) workflows, including data analysis, feature engineering, model training, and competition…

Artificial Intelligence · Computer Science 2025-09-12 Hangyi Jia , Yuxi Qian , Hanwen Tong , Xinhui Wu , Lin Chen , Feng Wei

Heterogeneous hardware and dynamic workloads worsen long-standing OS bottlenecks in scalability, adaptability, and manageability. At the same time, advances in machine learning (ML), large language models (LLMs), and agent-based methods…

Operating Systems · Computer Science 2025-11-12 Yifan Zhang , Xinkui Zhao , Ziying Li , Guanjie Cheng , Jianwei Yin , Lufei Zhang , Zuoning Chen

With the remarkable advancements of large language models (LLMs), LLM-based agents have become a research hotspot in human-computer interaction. However, there is a scarcity of benchmarks available for LLM-based mobile agents. Benchmarking…

Artificial Intelligence · Computer Science 2024-07-02 Shihan Deng , Weikai Xu , Hongda Sun , Wei Liu , Tao Tan , Jianfeng Liu , Ang Li , Jian Luan , Bin Wang , Rui Yan , Shuo Shang