DyFo：一种无训练的动态聚焦视觉搜索方法用于提升大型多模态模型在细粒度视觉理解中的性能
摘要
人类能够轻松地在杂乱环境中定位所需对象，依赖一种称为视觉搜索的认知机制，高效地过滤无关信息并聚焦于任务相关区域。 Inspired by this process, 我们提出Dyfo（Dynamic Focus），一种基于无训练动态聚焦视觉搜索的方法，用于提升大型多模态模型（LMMs）在细粒度视觉理解方面的性能。 Unlike existing approaches which require additional modules or data collection, Dyfo leverages a bidirectional interaction between LMMs and visual experts, using a Monte Carlo Tree Search (MCTS) algorithm to simulate human-like focus adjustments. This enables LMMs to focus on key visual regions while filtering out irrelevant content, without introducing additional training caused by vocabulary expansion or the integration of specialized localization modules. Experimental results demonstrate that Dyfo significantly improves fine-grained visual understanding and reduces hallucination issues in LMMs, achieving superior performance across both fixed and dynamic resolution models. The code is available at https://github.com/PKU-ICST-MIPL/DyFo_CVPR2025
引用
@article{arxiv.2504.14920,
title = {DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understanding},
author = {Geng Li and Jinglin Xu and Yunzhen Zhao and Yuxin Peng},
journal= {arXiv preprint arXiv:2504.14920},
year = {2025}
}
备注
Accepted by CVPR 2025 (Hightlight). Project page with code: https://github.com/PKU-ICST-MIPL/DyFo_CVPR2025