We share our experience with the recently released WILDS benchmark, a collection of ten datasets dedicated to developing models and training strategies which are robust to domain shifts. Several experiments yield a couple of critical observations which we believe are of general interest for any future work on WILDS. Our study focuses on two datasets: iWildCam and FMoW. We show that (1) Conducting separate cross-validation for each evaluation metric is crucial for both datasets, (2) A weak correlation between validation and test performance might make model development difficult for iWildCam, (3) Minor changes in the training of hyper-parameters improve the baseline by a relatively large margin (mainly on FMoW), (4) There is a strong correlation between certain domains and certain target labels (mainly on iWildCam). To the best of our knowledge, no prior work on these datasets has reported these observations despite their obvious importance. Our code is public.
@article{arxiv.2112.15550,
title = {Improving Baselines in the Wild},
author = {Kazuki Irie and Imanol Schlag and Róbert Csordás and Jürgen Schmidhuber},
journal= {arXiv preprint arXiv:2112.15550},
year = {2022}
}
Comments
Presented at NeurIPS 2021 Workshop on Distribution Shifts, https://openreview.net/forum?id=9vxOrkNTs1x