Deep Learning without Weight Transport
Abstract
Current algorithms for deep learning probably cannot run in the brain because they rely on weight transport, where forward-path neurons transmit their synaptic weights to a feedback path, in a way that is likely impossible biologically. An algorithm called feedback alignment achieves deep learning without weight transport by using random feedback weights, but it performs poorly on hard visual-recognition tasks. Here we describe two mechanisms - a neural circuit called a weight mirror and a modification of an algorithm proposed by Kolen and Pollack in 1994 - both of which let the feedback path learn appropriate synaptic weights quickly and accurately even in large networks, without weight transport or complex wiring.Tested on the ImageNet visual-recognition task, these mechanisms outperform both feedback alignment and the newer sign-symmetry method, and nearly match backprop, the standard algorithm of deep learning, which uses weight transport.
Cite
@article{arxiv.1904.05391,
title = {Deep Learning without Weight Transport},
author = {Mohamed Akrout and Collin Wilson and Peter C. Humphreys and Timothy Lillicrap and Douglas Tweed},
journal= {arXiv preprint arXiv:1904.05391},
year = {2020}
}
Comments
Accepted for the Conference on Neural Information Processing Systems (NeurIPS) 2019