English
Related papers

Related papers: Gym-Anything: Turn any Software into an Agent Envi…

200 papers

Computer-Use Agents (CUAs) are an increasingly deployed class of agents that take actions on GUIs to accomplish user goals. In this paper, we show that CUAs consistently exhibit Blind Goal-Directedness (BGD): a bias to pursue goals…

The recent progress of large language model agents has opened new possibilities for automating tasks through graphical user interfaces (GUIs), especially in mobile environments where intelligent interaction can greatly enhance usability.…

In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and…

Goal-directed interactive agents, which autonomously complete tasks through interactions with their environment, can assist humans in various domains of their daily lives. Recent advances in large language models (LLMs) led to a surge of…

Computation and Language · Computer Science 2024-09-30 Mareike Hartmann , Alexander Koller

Modern agents built on frontier language models often cannot adapt their weights. What, then, remains trainable? We argue it is the agent's \emph{workspace}, the structured external substrate it reads, writes, and tests; we call its…

Artificial Intelligence · Computer Science 2026-05-12 Elad Sarafian , Gal Kaplun , Ron Banner , Daniel Soudry , Boris Ginsburg

Many modern cyber physical systems incorporate computer vision technologies, complex sensors and advanced control software, allowing them to interact with the environment autonomously. Testing such systems poses numerous challenges: not…

Neural and Evolutionary Computing · Computer Science 2022-03-24 Dmytro Humeniuk , Foutse Khomh , Giuliano Antoniol

The growing capabilities of large language models (LLMs) in instruction-following and context-understanding lead to the era of agents with numerous applications. Among these, task planning agents have become especially prominent in…

Computer-use agents face a fundamental limitation. They rely exclusively on primitive GUI actions (click, type, scroll), creating brittle execution chains prone to cascading failures. While API-driven agents harness rich capabilities…

Computer Vision and Pattern Recognition · Computer Science 2026-05-27 Yuhao Yang , Zhen Yang , Zi-Yi Dou , Anh Nguyen , Keen You , Omar Attia , Andrew Szot , Michael Feng , Ram Ramrakhya , Alexander Toshev , Chao Huang , Yinfei Yang , Zhe Gan

As Large Language Models (LLMs) become ubiquitous across various scientific domains, their lack of ability to perform complex tasks like running simulations or to make complex decisions limits their utility. LLM-based agents bridge this gap…

Computation and Language · Computer Science 2026-01-21 Anurag Acharya , Timothy Vega , Rizwan A. Ashraf , Anshu Sharma , Derek Parker , Robert Rallo

With the rise of large language models (LLMs), LLM agents capable of autonomous reasoning, planning, and executing complex tasks have become a frontier in artificial intelligence. However, how to translate the research on general agents…

While LLM/VLM-powered AI agents have advanced rapidly in math, coding, and computer use, their applications in complex physical and social environments remain challenging. Building agents that can survive and thrive in the real world (for…

Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models…

Computer Vision and Pattern Recognition · Computer Science 2026-05-13 Miaosen Zhang , Xiaohan Zhao , Zhihong Tan , Zhou Huoshen , Yijia Fan , Yifan Yang , Kai Qiu , Bei Liu , Justin Wagle , Chenzhong Yin , Mingxi Cheng , Ji Li , Qi Dai , Chong Luo , Xu Yang , Xin Geng , Baining Guo

Interest in applying Artificial Intelligence (AI) techniques to compiler optimizations is increasing rapidly, but compiler research has a high entry barrier. Unlike in other domains, compiler and AI researchers do not have access to the…

Agents based on large language models leverage tools to modify environments, revolutionizing how AI interacts with the physical world. Unlike traditional NLP tasks that rely solely on historical dialogue for responses, these agents must…

Artificial Intelligence · Computer Science 2025-06-30 Peijie Yu , Yifan Yang , Jinjian Li , Zelong Zhang , Haorui Wang , Xiao Feng , Feng Zhang

Motivational digital systems offer capabilities to engage and motivate end-users to foster behavioral changes towards a common goal. In general these systems use gamification principles in non-games contexts. Over the years, gamification…

Artificial Intelligence · Computer Science 2021-03-30 Antonio Bucchiarone , Antonio Cicchetti , Nelly Bencomo , Enrica Loria , Annapaola Marconi

Graphical User Interface (GUI) agents adopt an end-to-end paradigm that maps a screenshot to an action sequence, thereby automating repetitive tasks in virtual environments. However, existing GUI agents are evaluated almost exclusively on…

Computer Vision and Pattern Recognition · Computer Science 2025-12-15 Chunyi Li , Longfei Li , Zicheng Zhang , Xiaohong Liu , Min Tang , Weisi Lin , Guangtao Zhai

Training AI models has always been challenging, especially when there is a need for custom models to provide personalized services. Algorithm engineers often face a lengthy process to iteratively develop models tailored to specific business…

Artificial Intelligence · Computer Science 2023-11-27 Haoyuan Li , Hao Jiang , Tianke Zhang , Zhelun Yu , Aoxiong Yin , Hao Cheng , Siming Fu , Yuhao Zhang , Wanggui He

Advanced agentic intelligence is a prerequisite for deploying Large Language Models in practical, real-world applications. Diverse real-world APIs demand precise, robust function-calling intelligence, which needs agents to develop these…

We introduce OceanGym, the first comprehensive benchmark for ocean underwater embodied agents, designed to advance AI in one of the most demanding real-world environments. Unlike terrestrial or aerial domains, underwater settings present…

Computation and Language · Computer Science 2025-11-26 Yida Xue , Mingjun Mao , Xiangyuan Ru , Yuqi Zhu , Baochang Ren , Shuofei Qiao , Mengru Wang , Shumin Deng , Xinyu An , Ningyu Zhang , Ying Chen , Huajun Chen

AI coding agents are increasingly used to write real-world software, but ensuring that their outputs are correct remains a fundamental challenge. Formal verification offers a promising path: an agent generates code together with a…