Non-Determinism in TensorFlow ResNets
Machine Learning
2020-01-31 v1 Neural and Evolutionary Computing
Machine Learning
Abstract
We show that the stochasticity in training ResNets for image classification on GPUs in TensorFlow is dominated by the non-determinism from GPUs, rather than by the initialisation of the weights and biases of the network or by the sequence of minibatches given. The standard deviation of test set accuracy is 0.02 with fixed seeds, compared to 0.027 with different seeds---nearly 74\% of the standard deviation of a ResNet model is non-deterministic. For test set loss the ratio of standard deviations is more than 80\%. These results call for more robust evaluation strategies of deep learning models, as a significant amount of the variation in results across runs can arise simply from GPU randomness.
Cite
@article{arxiv.2001.11396,
title = {Non-Determinism in TensorFlow ResNets},
author = {Miguel Morin and Matthew Willetts},
journal= {arXiv preprint arXiv:2001.11396},
year = {2020}
}
Comments
4 pages