The geometry of the deep linear network
Neural and Evolutionary Computing
2024-11-15 v1 Dynamical Systems
Probability
Adaptation and Self-Organizing Systems
Abstract
This article provides an expository account of training dynamics in the Deep Linear Network (DLN) from the perspective of the geometric theory of dynamical systems. Rigorous results by several authors are unified into a thermodynamic framework for deep learning. The analysis begins with a characterization of the invariant manifolds and Riemannian geometry in the DLN. This is followed by exact formulas for a Boltzmann entropy, as well as stochastic gradient descent of free energy using a Riemannian Langevin Equation. Several links between the DLN and other areas of mathematics are discussed, along with some open questions.
Cite
@article{arxiv.2411.09004,
title = {The geometry of the deep linear network},
author = {Govind Menon},
journal= {arXiv preprint arXiv:2411.09004},
year = {2024}
}