This technical report outlines the methodologies we applied for the PRCV Challenge, focusing on cognition and decision-making in driving scenarios. We employed InternVL-2.0, a pioneering open-source multi-modal model, and enhanced it by refining both the model input and training methodologies. For the input data, we strategically concatenated and formatted the multi-view images. It is worth mentioning that we utilized the coordinates of the original images without transformation. In terms of model training, we initially pre-trained the model on publicly available autonomous driving scenario datasets to bolster its alignment capabilities of the challenge tasks, followed by fine-tuning on the DriveLM-nuscenes Dataset. During the fine-tuning phase, we innovatively modified the loss function to enhance the model's precision in predicting coordinate values. These approaches ensure that our model possesses advanced cognitive and decision-making capabilities in driving scenarios. Consequently, our model achieved a score of 0.6064, securing the first prize on the competition's final results.
@article{arxiv.2411.02999,
title = {Precise Drive with VLM: First Prize Solution for PRCV 2024 Drive LM challenge},
author = {Bin Huang and Siyu Wang and Yuanpeng Chen and Yidan Wu and Hui Song and Zifan Ding and Jing Leng and Chengpeng Liang and Peng Xue and Junliang Zhang and Tiankun Zhao},
journal= {arXiv preprint arXiv:2411.02999},
year = {2024}
}