Redundancy in Deep Linear Neural Networks
Machine Learning
2022-06-10 v1 Performance
Abstract
Conventional wisdom states that deep linear neural networks benefit from expressiveness and optimization advantages over a single linear layer. This paper suggests that, in practice, the training process of deep linear fully-connected networks using conventional optimizers is convex in the same manner as a single linear fully-connected layer. This paper aims to explain this claim and demonstrate it. Even though convolutional networks are not aligned with this description, this work aims to attain a new conceptual understanding of fully-connected linear networks that might shed light on the possible constraints of convolutional settings and non-linear architectures.
Cite
@article{arxiv.2206.04490,
title = {Redundancy in Deep Linear Neural Networks},
author = {Oriel BenShmuel},
journal= {arXiv preprint arXiv:2206.04490},
year = {2022}
}