An initial alignment between neural network and target is needed for gradient descent to learn
Machine Learning
2022-08-17 v2
Abstract
This paper introduces the notion of ``Initial Alignment'' (INAL) between a neural network at initialization and a target function. It is proved that if a network and a Boolean target function do not have a noticeable INAL, then noisy gradient descent on a fully connected network with normalized i.i.d. initialization will not learn in polynomial time. Thus a certain amount of knowledge about the target (measured by the INAL) is needed in the architecture design. This also provides an answer to an open problem posed in [AS20]. The results are based on deriving lower-bounds for descent algorithms on symmetric neural networks without explicit knowledge of the target function beyond its INAL.
Keywords
Cite
@article{arxiv.2202.12846,
title = {An initial alignment between neural network and target is needed for gradient descent to learn},
author = {Emmanuel Abbe and Elisabetta Cornacchia and Jan Hązła and Christopher Marquis},
journal= {arXiv preprint arXiv:2202.12846},
year = {2022}
}