English
Related papers

Related papers: OS-Marathon: Benchmarking Computer-Use Agents on L…

200 papers

For decades, human-computer interaction has fundamentally been manual. Even today, almost all productive work done on the computer necessitates human input at every step. Autonomous virtual agents represent an exciting step in automating…

Artificial Intelligence · Computer Science 2024-07-23 Raghav Kapoor , Yash Parag Butala , Melisa Russak , Jing Yu Koh , Kiran Kamble , Waseem Alshikh , Ruslan Salakhutdinov

Longitudinal health agents must reason across multi-source trajectories that combine continuous device streams, sparse clinical exams, and episodic life events - yet evaluating them is hard: real-world data cannot be released at scale, and…

Artificial Intelligence · Computer Science 2026-04-06 Chao Li , Cailiang Liu , Ang Gao , Kexin Deng , Shu Zhang , Langping Xu , Xiaotong Shi , Xionghao Ding , Jian Pei , Xun Jiang

AI agents are an exciting new research direction, and agent development is driven by benchmarks. Our analysis of current agent benchmarks and evaluation practices reveals several shortcomings that hinder their usefulness in real-world…

Machine Learning · Computer Science 2024-07-02 Sayash Kapoor , Benedikt Stroebl , Zachary S. Siegel , Nitya Nadgir , Arvind Narayanan

Recent years, multimodal models have made remarkable strides and pave the way for intelligent browser use agents. However, when solving tasks on real world webpages in multi-turn, long-horizon trajectories, current agents still suffer from…

Artificial Intelligence · Computer Science 2025-09-26 Kaiwen He , Zhiwei Wang , Chenyi Zhuang , Jinjie Gu

Real-world multimodal agents solve multi-step workflows grounded in visual evidence. For example, an agent can troubleshoot a device by linking a wiring photo to a schematic and validating the fix with online documentation, or plan a trip…

Computer Vision and Pattern Recognition · Computer Science 2026-03-03 Zhaochen Su , Jincheng Gao , Hangyu Guo , Zhenhua Liu , Lueyang Zhang , Xinyu Geng , Shijue Huang , Peng Xia , Guanyu Jiang , Cheng Wang , Yue Zhang , Yi R. Fung , Junxian He

Language model (LM)-based agents have demonstrated promising capabilities in automating complex tasks from natural language instructions, yet they continue to struggle with long-horizon planning and reasoning. To address this, we propose an…

Artificial Intelligence · Computer Science 2026-05-05 Wenyi Wu , Sibo Zhu , Kun Zhou , Biwei Huang

AI agents have been developed for complex real-world tasks from coding to customer service. But AI agent evaluations suffer from many challenges that undermine our understanding of how well agents really work. We introduce the Holistic…

As large language models (LLMs) evolve into sophisticated autonomous agents capable of complex software development tasks, evaluating their real-world capabilities becomes critical. While existing benchmarks like…

Interactive agent benchmarks face a tension between scalable construction and realistic workflow evaluation. Hand-authored tasks are expensive to extend and revise, while static prompt evaluation misses failures that only appear when agents…

Artificial Intelligence · Computer Science 2026-05-19 Yuxiang Lai , Peng Xia , Haonian Ji , Kaiwen Xiong , Kaide Zeng , Jiaqi Liu , Fang Wu , Jike Zhong , Zeyu Zheng , Cihang Xie , Huaxiu Yao

Large language model (LLM) agents are increasingly used to assist with operations research (OR) modeling, yet existing OR-oriented benchmarks often reduce evaluation to one-shot translation from a self-contained problem statement into a…

Artificial Intelligence · Computer Science 2026-05-28 Chenyu Zhou , Xinyun Lu , Jiangyue Zhao , Jianghao Lin , Dongdong Ge , Yinyu Ye

Benchmarks for large language models (LLMs) have predominantly assessed short-horizon, localized reasoning. Existing long-horizon suites (e.g. SWE-bench) rely on manually curated issues, so expanding or tuning difficulty demands expensive…

Machine Learning · Computer Science 2025-06-03 Kaivalya Hariharan , Uzay Girit , Atticus Wang , Jacob Andreas

Computer-using agents (CUAs) enable task completion through natural interaction with operating systems and software interfaces. While script-based verifiers are widely adopted for evaluation, they suffer from limited scalability and…

Software Engineering · Computer Science 2025-10-22 Haojia Lin , Xiaoyu Tan , Yulei Qin , Zihan Xu , Yuchen Shi , Zongyi Li , Gang Li , Shaofei Cai , Siqi Cai , Chaoyou Fu , Ke Li , Xing Sun

Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely overlook agents' ability to orchestrate tools for such rigorous workflows. To bridge this gap,…

LLM-based agents have demonstrated strong capabilities in solving complex tasks through multi-step reasoning and tool use. However, existing evaluation protocols primarily focus on task success, overlooking a critical aspect of agent…

Artificial Intelligence · Computer Science 2026-05-29 Minyang Hu , Bo Yang , Zhinuo Zhou , Jiachen Liang , Guo Jiahao , Yiyang Yin , Xiongwei Han

Language agents powered by large language models (LLMs) have demonstrated remarkable capabilities in understanding, reasoning, and executing complex tasks. However, developing robust agents presents significant challenges: substantial…

Computation and Language · Computer Science 2025-06-02 Qianqian Zhang , Jiajia Liao , Heting Ying , Yibo Ma , Haozhan Shen , Jingcheng Li , Peng Liu , Lu Zhang , Chunxin Fang , Kyusong Lee , Ruochen Xu , Tiancheng Zhao

Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address…

Periodic human activities with implicit workflows are common in manufacturing, sports, and daily life. While short-term periodic activities -- characterized by simple structures and high-contrast patterns -- have been widely studied,…

Computer Vision and Pattern Recognition · Computer Science 2025-11-21 Fan Yang , Quanting Xie , Atsunori Moteki , Shoichi Masui , Shan Jiang , Kanji Uchino , Yonatan Bisk , Graham Neubig

To fulfill user instructions, autonomous web agents must contend with the inherent complexity and volatile nature of real-world websites. Conventional paradigms predominantly rely on Supervised Fine-Tuning (SFT) or Offline Reinforcement…

Artificial Intelligence · Computer Science 2026-05-01 Yuyu Guo , Wenjie Yang , Siyuan Yang , Ziyang Liu , Cheng Chen , Yuan Wei , Yun Hu , Yang Huang , Guoliang Hao , Dongsheng Yuan , Jianming Wang , Xin Chen , Hang Yu , Lei Lei , Peng Di

Effectively processing long contexts remains a fundamental yet unsolved challenge for large language models (LLMs). Existing single-LLM-based methods primarily reduce the context window or optimize the attention mechanism, but they often…

Computation and Language · Computer Science 2026-04-22 Yichen Jiang , Jiakang Yuan , Chongjun Tu , Peng Ye , Tao Chen

Computer-Using Agents (CUAs) aim to autonomously operate computer systems to complete real-world tasks. However, existing agentic systems remain difficult to scale and lag behind human performance. A key limitation is the absence of…