Multimodal image-text contrastive learning has shown that joint representations can be learned across modalities. Here, we show how leveraging multiple views of image data with contrastive learning can improve downstream fine-grained classification performance for species recognition, even when one view is absent. We propose ContRastive Image-remote Sensing Pre-training (CRISP)\unicodex2014a new pre-training task for ground-level and aerial image representation learning of the natural world\unicodex2014and introduce Nature Multi-View (NMV), a dataset of natural world imagery including >3 million ground-level and aerial image pairs for over 6,000 plant taxa across the ecologically diverse state of California. The NMV dataset and accompanying material are available at hf.co/datasets/andyvhuynh/NatureMultiView.
@article{arxiv.2409.19439,
title = {Contrastive ground-level image and remote sensing pre-training improves representation learning for natural world imagery},
author = {Andy V. Huynh and Lauren E. Gillespie and Jael Lopez-Saucedo and Claire Tang and Rohan Sikand and Moisés Expósito-Alonso},
journal= {arXiv preprint arXiv:2409.19439},
year = {2024}
}