In light of the dynamic nature of autonomous driving environments and stringent safety requirements, general MLLMs combined with CLIP alone often struggle to accurately represent driving-specific scenarios, particularly in complex interactions and long-tail cases. To address this, we propose the Hints of Prompt (HoP) framework, which introduces three key enhancements: Affinity hint to emphasize instance-level structure by strengthening token-wise connections, Semantic hint to incorporate high-level information relevant to driving-specific cases, such as complex interactions among vehicles and traffic signs, and Question hint to align visual features with the query context, focusing on question-relevant regions. These hints are fused through a Hint Fusion module, enriching visual representations by capturing driving-related representations with limited domain data, ensuring faster adaptation to driving scenarios. Extensive experiments confirm the effectiveness of the HoP framework, showing that it significantly outperforms previous state-of-the-art methods in all key metrics.
Cite
@article{arxiv.2411.13076,
title = {Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Driving},
author = {Hao Zhou and Zhanning Gao and Zhili Chen and Maosheng Ye and Qifeng Chen and Tongyi Cao and Honggang Qi},
journal= {arXiv preprint arXiv:2411.13076},
year = {2025}
}