How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?
Abstract
It is widely believed that the practical success of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) owes to the fact that CNNs and RNNs use a more compact parametric representation than their Fully-Connected Neural Network (FNN) counterparts, and consequently require fewer training examples to accurately estimate their parameters. We initiate the study of rigorously characterizing the sample-complexity of estimating CNNs and RNNs. We show that the sample-complexity to learn CNNs and RNNs scales linearly with their intrinsic dimension and this sample-complexity is much smaller than for their FNN counterparts. For both CNNs and RNNs, we also present lower bounds showing our sample complexities are tight up to logarithmic factors. Our main technical tools for deriving these results are a localized empirical process analysis and a new technical lemma characterizing the convolutional and recurrent structure. We believe that these tools may inspire further developments in understanding CNNs and RNNs.
Cite
@article{arxiv.1805.07883,
title = {How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?},
author = {Simon S. Du and Yining Wang and Xiyu Zhai and Sivaraman Balakrishnan and Ruslan Salakhutdinov and Aarti Singh},
journal= {arXiv preprint arXiv:1805.07883},
year = {2019}
}
Comments
Revised version, with new results on recurrent neural networks. Preliminary version in NeurIPS 2018