English
Related papers

Related papers: InteractWeb-Bench: Can Multimodal Agent Escape Bli…

200 papers

The growing complexity of power systems has made accurate load forecasting more important than ever. An increasing number of advanced load forecasting methods have been developed. However, the static design of current methods offers no…

Machine Learning · Computer Science 2025-05-23 Yu Zuo , Dalin Qin , Yi Wang

AI agents are increasingly being deployed to automate tasks, often based on underspecified user instructions. Making unwarranted assumptions to compensate for the missing information and failing to ask clarifying questions can lead to…

Artificial Intelligence · Computer Science 2026-02-24 Sanidhya Vijayvargiya , Xuhui Zhou , Akhila Yerukola , Maarten Sap , Graham Neubig

With the rapid development of multimodal large language models (MLLMs), they are increasingly deployed as autonomous computer-use agents capable of accomplishing complex computer tasks. However, a pressing issue arises: Can the safety risk…

Artificial Intelligence · Computer Science 2025-06-23 Jingyi Yang , Shuai Shao , Dongrui Liu , Jing Shao

We introduce MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions, with a focus on open research problems that demand novel methodologies. Unlike…

Recent progress in GUI agents has substantially improved visual grounding, yet robust planning remains challenging, particularly when the environment deviates from a canonical initial state. In real applications, users often invoke…

Artificial Intelligence · Computer Science 2026-05-26 Henry Hengyuan Zhao , Kaiming Yang , Wendi Yu , Difei Gao , Mike Zheng Shou

Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing…

Despite the advancements and impressive performance of Multimodal Large Language Models (MLLMs) on benchmarks, their effectiveness in real-world, long-context, and multi-image tasks is unclear due to the benchmarks' limited scope. Existing…

Computation and Language · Computer Science 2024-05-16 Dingjie Song , Shunian Chen , Guiming Hardy Chen , Fei Yu , Xiang Wan , Benyou Wang

Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digital workflows. However, real-world execution environments are far from ideal: pop-ups,…

Artificial Intelligence · Computer Science 2026-05-26 Jingwei Sun , Jianing Zhu , Yuanyi Li , Tongliang Liu , Xia HU , Bo Han

Large Language Model (LLM)-based agentic systems have shown growing promise in tackling complex, multi-step tasks through autonomous planning, reasoning, and interaction with external environments. However, the stochastic nature of LLM…

Human-Computer Interaction · Computer Science 2026-03-31 Shuo Yan , Xiaolin Wen , Shaolun Ruan , Yanjie Zhang , Jiaming Mi , Yushi Sun , Huamin Qu , Rui Sheng

Recent advancements in Large Language Models (LLMs) and multimodal counterparts have spurred significant interest in developing web agents -- AI systems capable of autonomously navigating and completing tasks within web environments. While…

Machine Learning · Computer Science 2025-06-13 Xing Han Lù , Gaurav Kamath , Marius Mosbach , Siva Reddy

Exploratory GUI testing is essential for software quality but suffers from high manual costs. While Multi-modal Large Language Model (MLLM) agents excel in navigation, they fail to autonomously discover defects due to two core challenges:…

Artificial Intelligence · Computer Science 2026-01-09 Yifei Gao , Jiang Wu , Xiaoyi Chen , Yifan Yang , Zhe Cui , Tianyi Ma , Jiaming Zhang , Jitao Sang

Recent advances in large language models (LLMs) have shown impressive performance in mathematical reasoning and code generation. However, LLMs still struggle in the simulation domain, particularly in generating Simulink models, which are…

Machine Learning · Computer Science 2025-09-01 Xinxing Ren , Qianbo Zang , Zekun Guo

Front-end web code has become a core product surface for every frontier LLM release, yet evaluating these interactive applications at development speed remains costly because human-judged leaderboards like Arena do not scale. Existing…

Artificial Intelligence · Computer Science 2026-05-29 Haoyue Yang , Zhangxiao Shen , Fan Ding , Hangting Lou , Yifeng Kou , Haoqing Yu , Jingyao Li , Zhengfan Wu , Siqi Bao , Jing Liu , Hua Wu

Evaluating the performance of LLMs in multi-turn human-agent interactions presents significant challenges, particularly due to the complexity and variability of user behavior. In this paper, we introduce HammerBench, a novel benchmark…

Computation and Language · Computer Science 2025-02-18 Jun Wang , Jiamu Zhou , Muning Wen , Xiaoyun Mo , Haoyu Zhang , Qiqiang Lin , Cheng Jin , Xihuai Wang , Weinan Zhang , Qiuying Peng , Jun Wang

LLM-based user simulation is the primary mechanism for end-to-end agent evaluation, yet simulated users are poor proxies for real humans: unconstrained LLM defaults produce a Formalism Ceiling (style match rates of 6-8% against real users),…

Human-Computer Interaction · Computer Science 2026-05-21 Ming Zhu , Juntao Tan , Rithesh Murthy , Jielin Qiu , Liangwei Yang , Wenting Zhao , Silvio Savarese , Shelby Heinecke , Huan Wang

We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, macOS, Linux, iOS, Android, and Web platforms. It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation,…

With the rapid development of LLM-based agents, there is a growing trend to incorporate agent-specific data into the pre-training stage of LLMs, aiming to better align LLMs with real-world autonomous task execution. However, current…

Artificial Intelligence · Computer Science 2025-10-29 Jiarui Qin , Yunjia Xi , Junjie Huang , Renting Rui , Di Yin , Weiwen Liu , Yong Yu , Weinan Zhang , Xing Sun

Multi-agent LLM frameworks are widely used to accelerate the development of agent systems powered by large language models (LLMs). These frameworks impose distinct architectural structures that govern how agents interact, store information,…

Artificial Intelligence · Computer Science 2026-02-04 Abdelghny Orogat , Ana Rostam , Essam Mansour

The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks…

Recent advances in Vision-Language Models (VLMs) have enabled mobile agents to perceive and interact with real-world mobile environments based on human instructions. However, the current fully autonomous paradigm poses potential safety…

Artificial Intelligence · Computer Science 2026-04-28 Qihang Ai , Pi Bu , Yue Cao , Yingyao Wang , Jihao Gu , Jingxuan Xing , Zekun Zhu , Wei Jiang , Zhicheng Zheng , Jun Song , Yuning Jiang