The Geometric Structure of Fully-Connected ReLU Layers
Abstract
We formalize and interpret the geometric structure of -dimensional fully connected ReLU layers in neural networks. The parameters of a ReLU layer induce a natural partition of the input domain, such that the ReLU layer can be significantly simplified in each sector of the partition. This leads to a geometric interpretation of a ReLU layer as a projection onto a polyhedral cone followed by an affine transformation, in line with the description in [doi:10.48550/arXiv.1905.08922] for convolutional networks with ReLU activations. Further, this structure facilitates simplified expressions for preimages of the intersection between partition sectors and hyperplanes, which is useful when describing decision boundaries in a classification setting. We investigate this in detail for a feed-forward network with one hidden ReLU-layer, where we provide results on the geometric complexity of the decision boundary generated by such networks, as well as proving that modulo an affine transformation, such a network can only generate different decision boundaries. Finally, the effect of adding more layers to the network is discussed.
Keywords
Cite
@article{arxiv.2310.03482,
title = {The Geometric Structure of Fully-Connected ReLU Layers},
author = {Jonatan Vallin and Karl Larsson and Mats G. Larson},
journal= {arXiv preprint arXiv:2310.03482},
year = {2023}
}