Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
Abstract
Feed-forward neural networks consist of a sequence of layers, in which each layer performs some processing on the information from the previous layer. A downside to this approach is that each layer (or module, as multiple modules can operate in parallel) is tasked with processing the entire hidden state, rather than a particular part of the state which is most relevant for that module. Methods which only operate on a small number of input variables are an essential part of most programming languages, and they allow for improved modularity and code re-usability. Our proposed method, Neural Function Modules (NFM), aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm which, as we show, improves the results in standard classification, out-of-domain generalization, generative modeling, and learning representations in the context of reinforcement learning.
Cite
@article{arxiv.2010.08012,
title = {Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers},
author = {Alex Lamb and Anirudh Goyal and Agnieszka Słowik and Michael Mozer and Philippe Beaudoin and Yoshua Bengio},
journal= {arXiv preprint arXiv:2010.08012},
year = {2020}
}