Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks
Machine Learning
2024-05-29 v1 Optimization and Control
Abstract
This paper presents a comprehensive analysis of critical point sets in two-layer neural networks. To study such complex entities, we introduce the critical embedding operator and critical reduction operator as our tools. Given a critical point, we use these operators to uncover the whole underlying critical set representing the same output function, which exhibits a hierarchical structure. Furthermore, we prove existence of saddle branches for any critical set whose output function can be represented by a narrower network. Our results provide a solid foundation to the further study of optimization and training behavior of neural networks.
Keywords
Cite
@article{arxiv.2405.17501,
title = {Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks},
author = {Leyang Zhang and Yaoyu Zhang and Tao Luo},
journal= {arXiv preprint arXiv:2405.17501},
year = {2024}
}