On the Memorization Properties of Contrastive Learning
Machine Learning
2021-07-22 v1 Machine Learning
Abstract
Memorization studies of deep neural networks (DNNs) help to understand what patterns and how do DNNs learn, and motivate improvements to DNN training approaches. In this work, we investigate the memorization properties of SimCLR, a widely used contrastive self-supervised learning approach, and compare them to the memorization of supervised learning and random labels training. We find that both training objects and augmentations may have different complexity in the sense of how SimCLR learns them. Moreover, we show that SimCLR is similar to random labels training in terms of the distribution of training objects complexity.
Cite
@article{arxiv.2107.10143,
title = {On the Memorization Properties of Contrastive Learning},
author = {Ildus Sadrtdinov and Nadezhda Chirkova and Ekaterina Lobacheva},
journal= {arXiv preprint arXiv:2107.10143},
year = {2021}
}
Comments
Published in Workshop on Overparameterization: Pitfalls & Opportunities at ICML 2021