Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
Machine Learning
2021-07-05 v2
Abstract
We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite width network al-lows data dependent feature learning in its bottle-neck representation. We empirically show that a single bottleneck in infinite networks dramatically accelerates training when compared to purely in-finite networks, with an improved overall performance. We discuss the acceleration phenomena by drawing similarities to infinitely wide deep linear models, where the acceleration effect of a bottleneck can be understood theoretically.
Cite
@article{arxiv.2107.00364,
title = {Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks},
author = {Etai Littwin and Omid Saremi and Shuangfei Zhai and Vimal Thilak and Hanlin Goh and Joshua M. Susskind and Greg Yang},
journal= {arXiv preprint arXiv:2107.00364},
year = {2021}
}