A lattice-based approach to the expressivity of deep ReLU neural networks

Vincent Corlay; Joseph J. Boutros; Philippe Ciblat; Loic Brunel

A lattice-based approach to the expressivity of deep ReLU neural networks

Machine Learning 2020-01-20 v2 Information Theory math.IT Machine Learning

Authors: Vincent Corlay , Joseph J. Boutros , Philippe Ciblat , Loic Brunel

Abstract

We present new families of continuous piecewise linear (CPWL) functions in Rn having a number of affine pieces growing exponentially in $n$ . We show that these functions can be seen as the high-dimensional generalization of the triangle wave function used by Telgarsky in 2016. We prove that they can be computed by ReLU networks with quadratic depth and linear width in the space dimension. We also investigate the approximation error of one of these functions by shallower networks and prove a separation result. The main difference between our functions and other constructions is their practical interest: they arise in the scope of channel coding. Hence, computing such functions amounts to performing a decoding operation.

Keywords

deep relu networks expressivity continuous piecewise linear functions channel coding approximation error separation

Cite

@article{arxiv.1902.11294,
  title  = {A lattice-based approach to the expressivity of deep ReLU neural networks},
  author = {Vincent Corlay and Joseph J. Boutros and Philippe Ciblat and Loic Brunel},
  journal= {arXiv preprint arXiv:1902.11294},
  year   = {2020}
}

Comments

arXiv admin note: text overlap with arXiv:1902.05146

A lattice-based approach to the expressivity of deep ReLU neural networks

Abstract

Keywords

Cite

Comments

Related papers