Approximation of Lipschitz Functions using Deep Spline Neural Networks
Abstract
Lipschitz-constrained neural networks have many applications in machine learning. Since designing and training expressive Lipschitz-constrained networks is very challenging, there is a need for improved methods and a better theoretical understanding. Unfortunately, it turns out that ReLU networks have provable disadvantages in this setting. Hence, we propose to use learnable spline activation functions with at least 3 linear regions instead. We prove that this choice is optimal among all component-wise -Lipschitz activation functions in the sense that no other weight constrained architecture can approximate a larger class of functions. Additionally, this choice is at least as expressive as the recently introduced non component-wise Groupsort activation function for spectral-norm-constrained weights. Previously published numerical results support our theoretical findings.
Cite
@article{arxiv.2204.06233,
title = {Approximation of Lipschitz Functions using Deep Spline Neural Networks},
author = {Sebastian Neumayer and Alexis Goujon and Pakshal Bohra and Michael Unser},
journal= {arXiv preprint arXiv:2204.06233},
year = {2022}
}