Poor starting points in machine learning
Machine Learning
2016-02-10 v1 Neural and Evolutionary Computing
Optimization and Control
Machine Learning
Abstract
Poor (even random) starting points for learning/training/optimization are common in machine learning. In many settings, the method of Robbins and Monro (online stochastic gradient descent) is known to be optimal for good starting points, but may not be optimal for poor starting points -- indeed, for poor starting points Nesterov acceleration can help during the initial iterations, even though Nesterov methods not designed for stochastic approximation could hurt during later iterations. The common practice of training with nontrivial minibatches enhances the advantage of Nesterov acceleration.
Cite
@article{arxiv.1602.02823,
title = {Poor starting points in machine learning},
author = {Mark Tygert},
journal= {arXiv preprint arXiv:1602.02823},
year = {2016}
}
Comments
11 pages, 3 figures, 1 table; this initial version is literally identical to that circulated among a restricted audience over a month ago