Transfer entropy-based feedback improves performance in artificial neural networks
Abstract
The structure of the majority of modern deep neural networks is characterized by uni- directional feed-forward connectivity across a very large number of layers. By contrast, the architecture of the cortex of vertebrates contains fewer hierarchical levels but many recurrent and feedback connections. Here we show that a small, few-layer artificial neural network that employs feedback will reach top level performance on a standard benchmark task, otherwise only obtained by large feed-forward structures. To achieve this we use feed-forward transfer entropy between neurons to structure feedback connectivity. Transfer entropy can here intuitively be understood as a measure for the relevance of certain pathways in the network, which are then amplified by feedback. Feedback may therefore be key for high network performance in small brain-like architectures.
Cite
@article{arxiv.1706.04265,
title = {Transfer entropy-based feedback improves performance in artificial neural networks},
author = {Sebastian Herzog and Christian Tetzlaff and Florentin Wörgötter},
journal= {arXiv preprint arXiv:1706.04265},
year = {2017}
}