We introduce a simple new regularizer for auto-encoders whose hidden-unit activation functions contain at least one zero-gradient (saturated) region. This regularizer explicitly encourages activations in the saturated region(s) of the corresponding activation function. We call these Saturating Auto-Encoders (SATAE). We show that the saturation regularizer explicitly limits the SATAE's ability to reconstruct inputs which are not near the data manifold. Furthermore, we show that a wide variety of features can be learned when different activation functions are used. Finally, connections are established with the Contractive and Sparse Auto-Encoders.
Cite
@article{arxiv.1301.3577,
title = {Saturating Auto-Encoders},
author = {Rostislav Goroshin and Yann LeCun},
journal= {arXiv preprint arXiv:1301.3577},
year = {2013}
}