As interest in "reformulating" the 3D Visual Question Answering (VQA) problem in the context of foundation models grows, it is imperative to assess how these new paradigms influence existing closed-vocabulary datasets. In this case study, we evaluate the zero-shot performance of foundational models (GPT-4 Vision and GPT-4) on well-established 3D VQA benchmarks, namely 3D-VQA and ScanQA. We provide an investigation to contextualize the performance of GPT-based agents relative to traditional modeling approaches. We find that GPT-based agents without any fine-tuning perform on par with the closed vocabulary approaches. Our findings corroborate recent results that "blind" models establish a surprisingly strong baseline in closed-vocabulary settings. We demonstrate that agents benefit significantly from scene-specific vocabulary via in-context textual grounding. By presenting a preliminary comparison with previous baselines, we hope to inform the community's ongoing efforts to refine multi-modal 3D benchmarks.
Cite
@article{arxiv.2405.18831,
title = {Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks},
author = {Simranjit Singh and Georgios Pavlakos and Dimitrios Stamoulis},
journal= {arXiv preprint arXiv:2405.18831},
year = {2024}
}
Comments
Accepted at 1st Workshop on Multimodalities for 3D Scenes CVPR 2024