On the Initialisation of Wide Low-Rank Feedforward Neural Networks
Machine Learning
2023-02-01 v1 Machine Learning
Abstract
The edge-of-chaos dynamics of wide randomly initialized low-rank feedforward networks are analyzed. Formulae for the optimal weight and bias variances are extended from the full-rank to low-rank setting and are shown to follow from multiplicative scaling. The principle second order effect, the variance of the input-output Jacobian, is derived and shown to increase as the rank to width ratio decreases. These results inform practitioners how to randomly initialize feedforward networks with a reduced number of learnable parameters while in the same ambient dimension, allowing reductions in the computational cost and memory constraints of the associated network.
Keywords
Cite
@article{arxiv.2301.13710,
title = {On the Initialisation of Wide Low-Rank Feedforward Neural Networks},
author = {Thiziri Nait Saada and Jared Tanner},
journal= {arXiv preprint arXiv:2301.13710},
year = {2023}
}