Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels

Weinan E; Stephan Wojtowytsch

Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels

Functional Analysis 2020-10-05 v2 Machine Learning Machine Learning

Authors: Weinan E , Stephan Wojtowytsch

Abstract

We establish a scale separation of Kolmogorov width type between subspaces of a given Banach space under the condition that a sequence of linear maps converges much faster on one of the subspaces. The general technique is then applied to show that reproducing kernel Hilbert spaces are poor $L^2$ -approximators for the class of two-layer neural networks in high dimension, and that multi-layer networks with small path norm are poor approximators for certain Lipschitz functions, also in the $L^2$ -topology.

Keywords

kolmogorov width banach space reproducing kernel hilbert space shallow neural networks random feature models neural tangent kernels l2-approximation

Cite

@article{arxiv.2005.10807,
  title  = {Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels},
  author = {Weinan E and Stephan Wojtowytsch},
  journal= {arXiv preprint arXiv:2005.10807},
  year   = {2020}
}

Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels

Abstract

Keywords

Cite

Related papers