Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
Machine Learning
2018-01-09 v4 Machine Learning
Abstract
Self-paced learning and hard example mining re-weight training instances to improve learning accuracy. This paper presents two improved alternatives based on lightweight estimates of sample uncertainty in stochastic gradient descent (SGD): the variance in predicted probability of the correct class across iterations of mini-batch SGD, and the proximity of the correct class probability to the decision threshold. Extensive experimental results on six datasets show that our methods reliably improve accuracy in various network architectures, including additional gains on top of other popular training techniques, such as residual learning, momentum, ADAM, batch normalization, dropout, and distillation.
Cite
@article{arxiv.1704.07433,
title = {Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples},
author = {Haw-Shiuan Chang and Erik Learned-Miller and Andrew McCallum},
journal= {arXiv preprint arXiv:1704.07433},
year = {2018}
}
Comments
camera-ready version for NIPS 2017