Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models
Abstract
The nonlinearity of activation functions used in deep learning models are crucial for the success of predictive models. There are several commonly used simple nonlinear functions, including Rectified Linear Unit (ReLU) and Leaky-ReLU (L-ReLU). In practice, these functions remarkably enhance the model accuracy. However, there is limited insight into the functionality of these nonlinear activation functions in terms of why certain models perform better than others. Here, we investigate the model performance when using ReLU or L-ReLU as activation functions in different model architectures and data domains. Interestingly, we found that the application of L-ReLU is mostly effective when the number of trainable parameters in a model is relatively small. Furthermore, we found that the image classification models seem to perform well with L-ReLU in fully connected layers, especially when pre-trained models such as the VGG-16 are used for the transfer learning.
Cite
@article{arxiv.2010.07359,
title = {Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models},
author = {Nalinda Kulathunga and Nishath Rajiv Ranasinghe and Daniel Vrinceanu and Zackary Kinsman and Lei Huang and Yunjiao Wang},
journal= {arXiv preprint arXiv:2010.07359},
year = {2020}
}
Comments
10 pages