Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
Abstract
The vast majority of deep models use multiple gradient signals, typically corresponding to a sum of multiple loss terms, to update a shared set of trainable weights. However, these multiple updates can impede optimal training by pulling the model in conflicting directions. We present Gradient Sign Dropout (GradDrop), a probabilistic masking procedure which samples gradients at an activation layer based on their level of consistency. GradDrop is implemented as a simple deep layer that can be used in any deep net and synergizes with other gradient balancing approaches. We show that GradDrop outperforms the state-of-the-art multiloss methods within traditional multitask and transfer learning settings, and we discuss how GradDrop reveals links between optimal multiloss training and gradient stochasticity.
Cite
@article{arxiv.2010.06808,
title = {Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout},
author = {Zhao Chen and Jiquan Ngiam and Yanping Huang and Thang Luong and Henrik Kretzschmar and Yuning Chai and Dragomir Anguelov},
journal= {arXiv preprint arXiv:2010.06808},
year = {2020}
}
Comments
Conference on Neural Information Processing Systems (NeurIPS) 2020