Practical recommendations for gradient-based training of deep architectures
Abstract
Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.
Cite
@article{arxiv.1206.5533,
title = {Practical recommendations for gradient-based training of deep architectures},
author = {Yoshua Bengio},
journal= {arXiv preprint arXiv:1206.5533},
year = {2012}
}