English
Related papers

Related papers: Toward Autonomous UI Exploration: The UIExplorer B…

200 papers

Smartphone GUI agents execute tasks by operating directly on app interfaces, offering a path to broad capability without deep system integration. However, real-world smartphone use is highly personalized: users adopt diverse workflows and…

Artificial Intelligence · Computer Science 2026-04-01 Hongyi Nie , Xunyuan Liu , Yudong Bai , Yaqing Wang , Yang Liu , Quanming Yao , Zhen Wang

Among existing online mobile-use benchmarks, AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation; however, recent agents achieving over 90% success rates indicate its…

Computation and Language · Computer Science 2026-01-01 Quyu Kong , Xu Zhang , Zhenyu Yang , Nolan Gao , Chen Liu , Panrong Tong , Chenglin Cai , Hanzhang Zhou , Jianan Zhang , Liangyu Chen , Zhidan Liu , Steven Hoi , Yue Wang

This paper introduces BioAgent Bench, a benchmark dataset and an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The benchmark contains curated end-to-end tasks (e.g.,…

Artificial Intelligence · Computer Science 2026-05-08 Dionizije Fa , Marko Culjak , Bruno Pandza , Mateo Cupic

User interface (UI) design goes beyond visuals to shape user experience (UX), underscoring the shift toward UI/UX as a unified concept. While recent studies have explored UI evaluation using Multimodal Large Language Models (MLLMs), they…

Computation and Language · Computer Science 2026-01-13 Jaehyun Jeon , Min Soo Kim , Jang Han Yoon , Sumin Shim , Yejin Choi , Hanbin Kim , Dae Hyun Kim , Youngjae Yu

The diversification of information access systems, from RAG to autonomous agents, creates a critical need for comparative user studies. However, the technical overhead to deploy and manage these distinct systems is a major barrier. We…

Human-Computer Interaction · Computer Science 2026-01-16 Saber Zerhoudi , Michael Granitzer

Automated web testing plays a critical role in ensuring high-quality user experiences and delivering business value. Traditional approaches primarily focus on code coverage and load testing, but often fall short of capturing complex user…

Software Engineering · Computer Science 2025-09-08 Naimeng Ye , Xiao Yu , Ruize Xu , Tianyi Peng , Zhou Yu

Autonomous web agents solve complex browsing tasks, yet existing benchmarks measure only whether an agent finishes a task, ignoring whether it does so safely or in a way enterprises can trust. To integrate these agents into critical…

Artificial Intelligence · Computer Science 2026-03-03 Ido Levy , Ben Wiesel , Sami Marreed , Alon Oved , Avi Yaeli , Segev Shlomov

Recent advances in mobile GUI agents have shown strong potential for automating mobile tasks, but most effective systems still depend on large vision-language models for screenshot understanding and long-horizon planning. Small GUI agents…

Artificial Intelligence · Computer Science 2026-05-29 Yuxiang Chai , Han Xiao , Xinyu Fu , Jinpeng Chen , Rui Liu , Hongsheng Li

Research in developmental psychology consistently shows that children explore the world thoroughly and efficiently and that this exploration allows them to learn. In turn, this early learning supports more robust generalization and…

Artificial Intelligence · Computer Science 2020-07-02 Eliza Kosoy , Jasmine Collins , David M. Chan , Sandy Huang , Deepak Pathak , Pulkit Agrawal , John Canny , Alison Gopnik , Jessica B. Hamrick

Contemporary GUI agents, while increasingly capable due to advances in Large Vision-Language Models (VLMs), often operate with a critical limitation: they treat each task in isolation, lacking a mechanism to systematically learn from past…

Artificial Intelligence · Computer Science 2026-04-13 Runze Li , Yuwen Zhai , Bo Xu , LiWu Xu , Nian Shi , Wei Zhang , Ran Lin , Liang Wang

Frontier coding agents are increasingly used in workflows where users supervise progress primarily through repeated improvement of a public score, namely the reported score on a public evaluation file with labels in the workspace, rather…

Computation and Language · Computer Science 2026-04-23 Hardy Chen , Nancy Lau , Haoqin Tu , Shuo Yan , Xiangyan Liu , Zijun Wang , Juncheng Wu , Michael Qizhe Shieh , Alvaro A. Cardenas , Cihang Xie , Yuyin Zhou

Graphical User Interface (GUI) agents show promising capabilities for automating computer-use tasks and facilitating accessibility, but existing interactive benchmarks are mostly English-only, covering web-use or Windows, Linux, and Android…

Artificial Intelligence · Computer Science 2025-10-21 Pei Yang , Hai Ci , Mike Zheng Shou

The development of autonomous agents for graphical user interfaces (GUIs) presents major challenges in artificial intelligence. While recent advances in native agent models have shown promise by unifying perception, reasoning, action, and…

Artificial Intelligence · Computer Science 2025-09-08 Haoming Wang , Haoyang Zou , Huatong Song , Jiazhan Feng , Junjie Fang , Junting Lu , Longxiang Liu , Qinyu Luo , Shihao Liang , Shijue Huang , Wanjun Zhong , Yining Ye , Yujia Qin , Yuwen Xiong , Yuxin Song , Zhiyong Wu , Aoyan Li , Bo Li , Chen Dun , Chong Liu , Daoguang Zan , Fuxing Leng , Hanbin Wang , Hao Yu , Haobin Chen , Hongyi Guo , Jing Su , Jingjia Huang , Kai Shen , Kaiyu Shi , Lin Yan , Peiyao Zhao , Pengfei Liu , Qinghao Ye , Renjie Zheng , Shulin Xin , Wayne Xin Zhao , Wen Heng , Wenhao Huang , Wenqian Wang , Xiaobo Qin , Yi Lin , Youbin Wu , Zehui Chen , Zihao Wang , Baoquan Zhong , Xinchun Zhang , Xujing Li , Yuanfan Li , Zhongkai Zhao , Chengquan Jiang , Faming Wu , Haotian Zhou , Jinlin Pang , Li Han , Qi Liu , Qianli Ma , Siyao Liu , Songhua Cai , Wenqi Fu , Xin Liu , Yaohui Wang , Zhi Zhang , Bo Zhou , Guoliang Li , Jiajun Shi , Jiale Yang , Jie Tang , Li Li , Qihua Han , Taoran Lu , Woyu Lin , Xiaokang Tong , Xinyao Li , Yichi Zhang , Yu Miao , Zhengxuan Jiang , Zili Li , Ziyuan Zhao , Chenxin Li , Dehua Ma , Feng Lin , Ge Zhang , Haihua Yang , Hangyu Guo , Hongda Zhu , Jiaheng Liu , Junda Du , Kai Cai , Kuanye Li , Lichen Yuan , Meilan Han , Minchao Wang , Shuyue Guo , Tianhao Cheng , Xiaobo Ma , Xiaojun Xiao , Xiaolong Huang , Xinjie Chen , Yidi Du , Yilin Chen , Yiwen Wang , Zhaojian Li , Zhenzhu Yang , Zhiyuan Zeng , Chaolin Jin , Chen Li , Hao Chen , Haoli Chen , Jian Chen , Qinghao Zhao , Guang Shi

Benchmarks are essential for unified evaluation and reproducibility. The rapid rise of Artificial Intelligence for Software Engineering (AI4SE) has produced numerous benchmarks for tasks such as code generation and bug repair. However, this…

Software Engineering · Computer Science 2025-12-15 Roham Koohestani , Philippe de Bekker , Begüm Koç , Maliheh Izadi

Recent advances in Multimodal Large Language Models (MLLMs) have enabled the development of mobile agents that can understand visual inputs and follow user instructions, unlocking new possibilities for automating complex tasks on mobile…

Robotics · Computer Science 2025-07-24 Ning Li , Xiangmou Qu , Jiamu Zhou , Jun Wang , Muning Wen , Kounianhua Du , Xingyu Lou , Qiuying Peng , Jun Wang , Weinan Zhang

Deep research agents integrate fragmented evidence through multi-step tool use. BrowseComp offers a text-only testbed for such agents, but existing multimodal benchmarks rarely require both weak visual cues composition and BrowseComp-style…

Computation and Language · Computer Science 2026-04-07 Xinyu Geng , Yanjing Xiao , Yuyang Zhang , Hanwen Wang , Xinyan Liu , Rui Min , Tianqing Fang , Yi R. Fung

Artificial intelligence systems are increasingly deployed in biomedical research. However, current evaluation frameworks may inadequately assess their effectiveness as research collaborators. This rapid review examines benchmarking…

Recent advancements in image generation models have enabled the prediction of future Graphical User Interface (GUI) states based on user instructions. However, existing benchmarks primarily focus on general domain visual fidelity, leaving…

Existing browser agent benchmarks face a fundamental trilemma: real-website benchmarks lack reproducibility due to content drift, controlled environments sacrifice realism by omitting real-web noise, and both require costly manual curation…

Artificial Intelligence · Computer Science 2026-04-14 Peng Yuan , Yuyang Yin , Yuxuan Cai , Zheng Wei

General virtual agents need to handle multimodal observations, master complex action spaces, and self-improve in dynamic, open-domain environments. However, existing environments are often domain-specific and require complex setups, which…

Artificial Intelligence · Computer Science 2025-02-17 Longtao Zheng , Zhiyuan Huang , Zhenghai Xue , Xinrun Wang , Bo An , Shuicheng Yan