Training Sparse Neural Networks

Suraj Srinivas; Akshayvarun Subramanya; R. Venkatesh Babu

Training Sparse Neural Networks

Computer Vision and Pattern Recognition 2016-11-22 v1 Machine Learning

Authors: Suraj Srinivas , Akshayvarun Subramanya , R. Venkatesh Babu

Abstract

Deep neural networks with lots of parameters are typically used for large-scale computer vision tasks such as image classification. This is a result of using dense matrix multiplications and convolutions. However, sparse computations are known to be much more efficient. In this work, we train and build neural networks which implicitly use sparse computations. We introduce additional gate variables to perform parameter selection and show that this is equivalent to using a spike-and-slab prior. We experimentally validate our method on both small and large networks and achieve state-of-the-art compression results for sparse neural network models.

Keywords

sparse learning deep neural networks neural network training

Cite

@article{arxiv.1611.06694,
  title  = {Training Sparse Neural Networks},
  author = {Suraj Srinivas and Akshayvarun Subramanya and R. Venkatesh Babu},
  journal= {arXiv preprint arXiv:1611.06694},
  year   = {2016}
}

Related papers

View all related →