Visualizing Information Bottleneck through Variational Inference
Machine Learning
2022-12-27 v1
Abstract
The Information Bottleneck theory provides a theoretical and computational framework for finding approximate minimum sufficient statistics. Analysis of the Stochastic Gradient Descent (SGD) training of a neural network on a toy problem has shown the existence of two phases, fitting and compression. In this work, we analyze the SGD training process of a Deep Neural Network on MNIST classification and confirm the existence of two phases of SGD training. We also propose a setup for estimating the mutual information for a Deep Neural Network through Variational Inference.
Cite
@article{arxiv.2212.12667,
title = {Visualizing Information Bottleneck through Variational Inference},
author = {Cipta Herwana and Abhishek Kadian},
journal= {arXiv preprint arXiv:2212.12667},
year = {2022}
}
Comments
arXiv admin note: text overlap with arXiv:1703.00810, arXiv:2202.06749 by other authors