A lot of recent advances in unsupervised feature learning are based on designing features which are invariant under semantic data augmentations. A common way to do this is contrastive learning, which uses positive and negative samples. Some recent works however have shown promising results for non-contrastive learning, which does not require negative samples. However, the non-contrastive losses have obvious "collapsed" minima, in which the encoders output a constant feature embedding, independent of the input. A folk conjecture is that so long as these collapsed solutions are avoided, the produced feature representations should be good. In our paper, we cast doubt on this story: we show through theoretical results and controlled experiments that even on simple data models, non-contrastive losses have a preponderance of non-collapsed bad minima. Moreover, we show that the training process does not avoid these minima.
@article{arxiv.2203.15702,
title = {Contrasting the landscape of contrastive and non-contrastive learning},
author = {Ashwini Pokle and Jinjin Tian and Yuchen Li and Andrej Risteski},
journal= {arXiv preprint arXiv:2203.15702},
year = {2022}
}
Comments
Accepted for publication in the AISTATS 2022 conference (http://aistats.org/aistats2022/accepted.html)