Learning Activation Functions to Improve Deep Neural Networks
Neural and Evolutionary Computing
2015-04-22 v3 Computer Vision and Pattern Recognition
Machine Learning
Machine Learning
Abstract
Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.
Cite
@article{arxiv.1412.6830,
title = {Learning Activation Functions to Improve Deep Neural Networks},
author = {Forest Agostinelli and Matthew Hoffman and Peter Sadowski and Pierre Baldi},
journal= {arXiv preprint arXiv:1412.6830},
year = {2015}
}
Comments
Accepted as a workshop paper contribution at the International Conference on Learning Representations (ICLR) 2015