中文

Trifuse:通过多模态融合增强基于注意力的 GUI 基准

人工智能 2026-02-09 v1 计算机视觉与模式识别

摘要

GUI grounding maps natural language instructions to the correct interface elements, serving as the perception foundation for GUI agents. Existing approaches predominantly rely on fine-tuning multimodal large language models (MLLMs) using large-scale GUI datasets to predict target element coordinates, which is data-intensive and generalizes poorly to unseen interfaces. Recent attention-based alternatives exploit localization signals in MLLMs attention mechanisms without task-specific fine-tuning, but suffer from low reliability due to the lack of explicit and complementary spatial anchors in GUI images. To address this limitation, we propose Trifuse, an attention-based grounding framework that explicitly integrates complementary spatial anchors. Trifuse integrates attention, OCR-derived textual cues, and icon-level caption semantics via a Consensus-SinglePeak (CS) fusion strategy that enforces cross-modal agreement while retaining sharp localization peaks. Extensive evaluations on four grounding benchmarks demonstrate that Trifuse achieves strong performance without task-specific fine-tuning, substantially reducing the reliance on expensive annotated data. Moreover, ablation studies reveal that incorporating OCR and caption cues consistently improves attention-based grounding performance across different backbones, highlighting its effectiveness as a general framework for GUI grounding.

关键词

引用

@article{arxiv.2602.06351,
  title  = {Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion},
  author = {Longhui Ma and Di Zhao and Siwei Wang and Zhao Lv and Miao Wang},
  journal= {arXiv preprint arXiv:2602.06351},
  year   = {2026}
}

备注

17 pages, 10 figures