Graphical User Interface (GUI) grounding - the task of mapping natural language instructions to screen coordinates - is essential for autonomous agents and accessibility technologies. Existing systems rely on monolithic models or one-shot pipelines that lack modularity and fail under visual clutter and ambiguous instructions. We introduce MEGA-GUI, a multi-stage framework that separates grounding into coarse Region-of-Interest (ROI) selection and fine-grained element grounding, orchestrated by specialized vision-language agents. MEGA-GUI features a bidirectional ROI zoom algorithm that mitigates spatial dilution and a context-aware rewriting agent that reduces semantic ambiguity. Our analysis reveals complementary strengths and weaknesses across vision-language models at different visual scales, and we show that leveraging this modular structure achieves consistently higher accuracy than monolithic approaches. On the visually dense ScreenSpot-Pro benchmark, MEGA-GUI attains 73.18% accuracy, and on the semantically complex OSWorld-G benchmark it reaches 68.63%, surpassing previously reported results. Code and the Grounding Benchmark Toolkit (GBT) are available at https://github.com/samsungsds-research-papers/mega-gui.
@article{arxiv.2511.13087,
title = {MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements},
author = {SeokJoo Kwak and Jihoon Kim and Boyoun Kim and Jung Jae Yoon and Wooseok Jang and Jeonghoon Hong and Jaeho Yang and Yeong-Dae Kwon},
journal= {arXiv preprint arXiv:2511.13087},
year = {2025}
}
Comments
26 pages, 7 figures. Code available at https://github.com/samsungsds-research-papers/mega-gui