Stochastic Neural Networks with Monotonic Activation Functions
Machine Learning
2016-07-25 v4 Machine Learning
Neural and Evolutionary Computing
Abstract
We propose a Laplace approximation that creates a stochastic unit from any smooth monotonic activation function, using only Gaussian noise. This paper investigates the application of this stochastic approximation in training a family of Restricted Boltzmann Machines (RBM) that are closely linked to Bregman divergences. This family, that we call exponential family RBM (Exp-RBM), is a subset of the exponential family Harmoniums that expresses family members through a choice of smooth monotonic non-linearity for each neuron. Using contrastive divergence along with our Gaussian approximation, we show that Exp-RBM can learn useful representations using novel stochastic units.
Cite
@article{arxiv.1601.00034,
title = {Stochastic Neural Networks with Monotonic Activation Functions},
author = {Siamak Ravanbakhsh and Barnabas Poczos and Jeff Schneider and Dale Schuurmans and Russell Greiner},
journal= {arXiv preprint arXiv:1601.00034},
year = {2016}
}
Comments
AISTATS 2016