De-fine: Decomposing and Refining Visual Programs with Auto-Feedback
Abstract
Visual programming, a modular and generalizable paradigm, integrates different modules and Python operators to solve various vision-language tasks. Unlike end-to-end models that need task-specific data, it advances in performing visual processing and reasoning in an unsupervised manner. Current visual programming methods generate programs in a single pass for each task where the ability to evaluate and optimize based on feedback, unfortunately, is lacking, which consequentially limits their effectiveness for complex, multi-step problems. Drawing inspiration from benders decomposition, we introduce De-fine, a training-free framework that automatically decomposes complex tasks into simpler subtasks and refines programs through auto-feedback. This model-agnostic approach can improve logical reasoning performance by integrating the strengths of multiple models. Our experiments across various visual tasks show that De-fine creates more robust programs. Moreover, viewing each feedback module as an independent agent will yield fresh prospects for the field of agent research.
Cite
@article{arxiv.2311.12890,
title = {De-fine: Decomposing and Refining Visual Programs with Auto-Feedback},
author = {Minghe Gao and Juncheng Li and Hao Fei and Liang Pang and Wei Ji and Guoming Wang and Zheqi Lv and Wenqiao Zhang and Siliang Tang and Yueting Zhuang},
journal= {arXiv preprint arXiv:2311.12890},
year = {2024}
}