Recent open-vocabulary detectors achieve promising performance with abundant region-level annotated data. In this work, we show that an open-vocabulary detector co-training with a large language model by generating image-level detailed captions for each image can further improve performance. To achieve the goal, we first collect a dataset, GroundingCap-1M, wherein each image is accompanied by associated grounding labels and an image-level detailed caption. With this dataset, we finetune an open-vocabulary detector with training objectives including a standard grounding loss and a caption generation loss. We take advantage of a large language model to generate both region-level short captions for each region of interest and image-level long captions for the whole image. Under the supervision of the large language model, the resulting detector, LLMDet, outperforms the baseline by a clear margin, enjoying superior open-vocabulary ability. Further, we show that the improved LLMDet can in turn build a stronger large multi-modal model, achieving mutual benefits. The code, model, and dataset is available at https://github.com/iSEE-Laboratory/LLMDet.
@article{arxiv.2501.18954,
title = {LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models},
author = {Shenghao Fu and Qize Yang and Qijie Mo and Junkai Yan and Xihan Wei and Jingke Meng and Xiaohua Xie and Wei-Shi Zheng},
journal= {arXiv preprint arXiv:2501.18954},
year = {2025}
}