Integrating information from multiple modalities enhances the robustness of scene perception systems in autonomous vehicles, providing a more comprehensive and reliable sensory framework. However, the modality incompleteness in multi-modal segmentation remains under-explored. In this work, we establish a task called Modality-Incomplete Scene Segmentation (MISS), which encompasses both system-level modality absence and sensor-level modality errors. To avoid the predominant modality reliance in multi-modal fusion, we introduce a Missing-aware Modal Switch (MMS) strategy to proactively manage missing modalities during training. Utilizing bit-level batch-wise sampling enhances the model's performance in both complete and incomplete testing scenarios. Furthermore, we introduce the Fourier Prompt Tuning (FPT) method to incorporate representative spectral information into a limited number of learnable prompts that maintain robustness against all MISS scenarios. Akin to fine-tuning effects but with fewer tunable parameters (1.1%). Extensive experiments prove the efficacy of our proposed approach, showcasing an improvement of 5.84% mIoU over the prior state-of-the-art parameter-efficient methods in modality missing. The source code is publicly available at https://github.com/RuipingL/MISS.
@article{arxiv.2401.16923,
title = {Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation},
author = {Ruiping Liu and Jiaming Zhang and Kunyu Peng and Yufan Chen and Ke Cao and Junwei Zheng and M. Saquib Sarfraz and Kailun Yang and Rainer Stiefelhagen},
journal= {arXiv preprint arXiv:2401.16923},
year = {2024}
}
Comments
Accepted to IEEE IV 2024. The source code is publicly available at https://github.com/RuipingL/MISS