Does Double Descent Occur in Self-Supervised Learning?
Machine Learning
2023-07-18 v1 Artificial Intelligence
Abstract
Most investigations into double descent have focused on supervised models while the few works studying self-supervised settings find a surprising lack of the phenomenon. These results imply that double descent may not exist in self-supervised models. We show this empirically using a standard and linear autoencoder, two previously unstudied settings. The test loss is found to have either a classical U-shape or to monotonically decrease instead of exhibiting a double-descent curve. We hope that further work on this will help elucidate the theoretical underpinnings of this phenomenon.
Cite
@article{arxiv.2307.07872,
title = {Does Double Descent Occur in Self-Supervised Learning?},
author = {Alisia Lupidi and Yonatan Gideoni and Dulhan Jayalath},
journal= {arXiv preprint arXiv:2307.07872},
year = {2023}
}
Comments
7 pages, 2 tables, 3 figures. Accepted for the workshop on High-Dimensional Learning Dynamics at ICML 2023