Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size. As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained environments using Kronecker product (KP). KPs can compress RNN layers by 16-38x with minimal accuracy loss. We show that KP can beat the task accuracy achieved by other state-of-the-art compression techniques (pruning and low-rank matrix factorization) across 4 benchmarks spanning 3 different applications, while simultaneously improving inference run-time.
@article{arxiv.1910.02558,
title = {Pushing the limits of RNN Compression},
author = {Urmish Thakker and Igor Fedorov and Jesse Beu and Dibakar Gope and Chu Zhou and Ganesh Dasika and Matthew Mattina},
journal= {arXiv preprint arXiv:1910.02558},
year = {2019}
}
Comments
6 pages. arXiv admin note: substantial text overlap with arXiv:1906.02876