Open-vocabulary object detection often fails under distribution shifts, as it can be misled by spurious correlations between non-causal visual attributes (e.g., brightness, texture) and object categories. Existing test-time adaptation (TTA) methods either depend on costly online optimization or perform global calibration, overlooking the attribute-specific nature of these failures. To address this, we propose FACTOR (counterFACtual training-free Test-time adaptation for Open-vocabulaRy object detection), a lightweight framework grounded in counterfactual reasoning. By perturbing test images along non-causal attributes and comparing region-level predictions between original and counterfactual views, FACTOR quantifies attribute sensitivity, semantic relevance, and prediction variation to selectively suppress attribute-dependent predictions-without parameter updates. Experiments on PASCAL-C, COCO-C, and FoggyCityscapes show that FACTOR consistently outperforms prior TTA methods, demonstrating that explicit counterfactual reasoning effectively improves robustness under distribution shifts.
@article{arxiv.2605.03294,
title = {FACTOR: Counterfactual Training-Free Test-Time Adaptation for Open-Vocabulary Object Detection},
author = {Kaixiang Zhao and Mao Ye and Lihua Zhou and Hu Wang and Luping Ji and Song Tang and Xiatian Zhu},
journal= {arXiv preprint arXiv:2605.03294},
year = {2026}
}