Deep orthogonal linear networks are shallow
Machine Learning
2020-11-30 v1 Machine Learning
Abstract
We consider the problem of training a deep orthogonal linear network, which consists of a product of orthogonal matrices, with no non-linearity in-between. We show that training the weights with Riemannian gradient descent is equivalent to training the whole factorization by gradient descent. This means that there is no effect of overparametrization and implicit bias at all in this setting: training such a deep, overparametrized, network is perfectly equivalent to training a one-layer shallow network.
Cite
@article{arxiv.2011.13831,
title = {Deep orthogonal linear networks are shallow},
author = {Pierre Ablin},
journal= {arXiv preprint arXiv:2011.13831},
year = {2020}
}