Learning deep representations by mutual information estimation and maximization
Abstract
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.
Cite
@article{arxiv.1808.06670,
title = {Learning deep representations by mutual information estimation and maximization},
author = {R Devon Hjelm and Alex Fedorov and Samuel Lavoie-Marchildon and Karan Grewal and Phil Bachman and Adam Trischler and Yoshua Bengio},
journal= {arXiv preprint arXiv:1808.06670},
year = {2019}
}
Comments
Accepted as an oral presentation at the International Conference for Learning Representations (ICLR), 2019