English
Related papers

Related papers: MMBench-GUI: Hierarchical Multi-Platform Evaluatio…

200 papers

In this report, we present ML-Dev-Bench, a benchmark aimed at testing agentic capabilities on applied Machine Learning development tasks. While existing benchmarks focus on isolated coding tasks or Kaggle-style competitions, ML-Dev-Bench…

Software Engineering · Computer Science 2025-02-20 Harshith Padigela , Chintan Shah , Dinkar Juyal

With the remarkable advancements of large language models (LLMs), LLM-based agents have become a research hotspot in human-computer interaction. However, there is a scarcity of benchmarks available for LLM-based mobile agents. Benchmarking…

Artificial Intelligence · Computer Science 2024-07-02 Shihan Deng , Weikai Xu , Hongda Sun , Wei Liu , Tao Tan , Jianfeng Liu , Ang Li , Jian Luan , Bin Wang , Rui Yan , Shuo Shang

The emergence of Multimodal Large Language Models (MLLMs) has driven significant advances in Graphical User Interface (GUI) agent capabilities. Nevertheless, existing GUI agent training and inference techniques still suffer from a dilemma…

Artificial Intelligence · Computer Science 2026-04-09 Shuquan Lian , Yuhang Wu , Jia Ma , Yifan Ding , Zihan Song , Bingqi Chen , Xiawu Zheng , Hui Li , Rongrong Ji

Developing intelligent agents capable of operating a wide range of Graphical User Interfaces (GUIs) with human-level proficiency is a key milestone on the path toward Artificial General Intelligence. While most existing datasets and…

Artificial Intelligence · Computer Science 2025-11-21 Sen Chen , Tong Zhao , Yi Bin , Fei Ma , Wenqi Shao , Zheng Wang

Embodied Planning is dedicated to the goal of creating agents capable of executing long-horizon tasks in complex physical worlds. However, existing embodied planning benchmarks frequently feature short-horizon tasks and coarse-grained…

Robotics · Computer Science 2025-08-06 Muzhen Cai , Xiubo Chen , Yining An , Jiaxin Zhang , Xuesong Wang , Wang Xu , Weinan Zhang , Ting Liu

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that…

Artificial Intelligence · Computer Science 2026-05-28 Yilun Yao , Xinyu Tan , Chao-Hsuan Liu , Yaoming Li , Zhengyang Wang , Wenhan Yu , Zhewen Tan , Yuxuan Tian , Guangxiang Zhao , Lin Sun , Xiangzheng Zhang , Tong Yang

Enterprise systems are crucial for enhancing productivity and decision-making among employees and customers. Integrating LLM based systems into enterprise systems enables intelligent automation, personalized experiences, and efficient…

Machine Learning · Computer Science 2025-11-03 Harsh Vishwakarma , Ankush Agarwal , Ojas Patil , Chaitanya Devaguptapu , Mahesh Chandran

Generative AI agents, software systems powered by Large Language Models (LLMs), are emerging as a promising approach to automate cybersecurity tasks. Among the others, penetration testing is a challenging field due to the task complexity…

Cryptography and Security · Computer Science 2024-10-29 Luca Gioacchini , Marco Mellia , Idilio Drago , Alexander Delsanto , Giuseppe Siracusano , Roberto Bifulco

GUI agents have emerged as a powerful paradigm for automating interactions in digital environments, yet achieving both broad generality and consistently strong task performance remains challenging. In this report, we present UI-Venus-1.5, a…

Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations, but this hybrid action space often leaves them uncertain about when to continue with…

Artificial Intelligence · Computer Science 2026-05-13 Xuhao Hu , Xi Zhang , Haiyang Xu , Kyle Qiao , Jingyi Yang , Xuanjing Huang , Jing Shao , Ming Yan , Jieping Ye

Foundation agents have rapidly advanced in their ability to reason and interact with real environments, making the evaluation of their core capabilities increasingly important. While many benchmarks have been developed to assess agent…

Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interaction. However, current evaluation…

Software Engineering · Computer Science 2026-05-29 Xiaoyi Chen , Yifei Gao , Yang Xu , Xingxing Song , Yi Zhang , Jitao Sang

The recent shift in Generative AI (GenAI) applications from cloud-only environments to end-user devices introduces new challenges in resource management, system efficiency, and user experience. This paper presents ConsumerBench, a…

Distributed, Parallel, and Cluster Computing · Computer Science 2025-06-24 Yile Gu , Rohan Kadekodi , Hoang Nguyen , Keisuke Kamahori , Yiyu Liu , Baris Kasikci

AI research agents accelerate ML research by automating hypothesis generation, experimentation, and empirical refinement. Existing agent strategies range from greedy hill-climbing to tree search and evolutionary optimization, yet which…

World models have shown great utility in improving the task performance of embodied agents. While prior work largely focuses on pixel-space world models, these approaches face practical limitations in GUI settings, where predicting complex…

Artificial Intelligence · Computer Science 2025-12-17 Shufan Li , Konstantinos Kallidromitis , Akash Gokul , Yusuke Kato , Kazuki Kozuka , Aditya Grover

Workspace learning requires AI agents to identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a worker's workspace, enabling them to complete both routine and advanced tasks effectively.…

Most web agents operate at the human interface level, observing screenshots or raw DOM trees without application-level access, which limits robustness and action expressiveness. In enterprise settings, however, explicit control of both the…

Artificial Intelligence · Computer Science 2026-02-17 Chenyang Ma , Clyde Fare , Matthew Wilson , Dave Braines

The performance and generalization of foundation models for interactive systems critically depend on the availability of large-scale, realistic training data. While recent advances in large language models (LLMs) have improved GUI…

Machine Learning · Computer Science 2026-03-25 Sofiya Garkot , Maksym Shamrai , Ivan Synytsia , Mariya Hirna

Large language models (LLMs) and agentic systems have shown promise for automated software development, but applying them to hardware-in-the-loop (HIL) embedded and Internet-of-Things (IoT) systems remains challenging due to the tight…

Software Engineering · Computer Science 2026-03-23 Yiming Li , Yuhan Cheng , Mingchen Ma , Yihang Zou , Ningyuan Yang , Wei Cheng , Hai "Helen" Li , Yiran Chen , Tingjun Chen

Graphical User Interface (GUI) grounding plays a crucial role in enhancing the capabilities of Vision-Language Model (VLM) agents. While general VLMs, such as GPT-4V, demonstrate strong performance across various tasks, their proficiency in…

Computer Vision and Pattern Recognition · Computer Science 2025-09-12 Anthony Nguyen