In this report, we relate the algorithmic design of Barlow Twins' method to the Hilbert-Schmidt Independence Criterion (HSIC), thus establishing it as a contrastive learning approach that is free of negative samples. Through this perspective, we argue that Barlow Twins (and thus the class of negative-sample-free contrastive learning methods) suggests a possibility to bridge the two major families of self-supervised learning philosophies: non-contrastive and contrastive approaches. In particular, Barlow twins exemplified how we could combine the best practices of both worlds: avoiding the need of large training batch size and negative sample pairing (like non-contrastive methods) and avoiding symmetry-breaking network designs (like contrastive methods).
Cite
@article{arxiv.2104.13712,
title = {A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning},
author = {Yao-Hung Hubert Tsai and Shaojie Bai and Louis-Philippe Morency and Ruslan Salakhutdinov},
journal= {arXiv preprint arXiv:2104.13712},
year = {2021}
}