Related papers: A multivariate Riesz basis of ReLU neural networks

Recently, Daubechies, DeVore, Foucart, Hanin, and Petrova introduced a system of piece-wise linear functions, which can be easily reproduced by artificial neural networks with the ReLU activation function and which form a Riesz basis of…

The seminal work of Daubechies, DeVore, Foucart, Hanin, and Petrova introduced in 2022 a sequence of univariate piece-wise linear functions, which resemble the classical Fourier basis and which, at the same time, can be easily reproduced by…

Scale invariance of an algorithm refers to its ability to treat objects equally independently of their size. For neural networks, scale invariance is typically achieved by data augmentation. However, when presented with a scale far outside…

We establish a functional large deviation principle for fully connected multi-layer perceptrons with i.i.d. Gaussian weights (LeCun initialization) and general Lipschitz activation functions, including therefore the popular case of ReLU.…

We prove that training neural networks on 1-D data is equivalent to solving convex Lasso problems with discrete, explicitly defined dictionary matrices. We consider neural networks with piecewise linear activations and depths ranging from 2…

Neural networks are playing a crucial role in everyday life, with the most modern generative models able to achieve impressive results. Nonetheless, their functioning is still not very clear, and several strategies have been adopted to…

Estimating the Riesz representer is central to debiased machine learning for causal and structural parameter estimation. We propose generalized Riesz regression, a unified framework for estimating the Riesz representer by fitting a…

Deep residual networks (ResNets) have demonstrated outstanding success in computer vision tasks, attributed to their ability to maintain gradient flow through deep architectures. Simultaneously, controlling the Lipschitz bound in neural…

In this study, we establish that deep neural networks employing ReLU and ReLU$^2$ activation functions can effectively represent Lagrange finite element functions of any order on various simplicial meshes in arbitrary dimensions. We…

We present new families of continuous piecewise linear (CPWL) functions in Rn having a number of affine pieces growing exponentially in $n$. We show that these functions can be seen as the high-dimensional generalization of the triangle…

Inspired by the work of Hedenmalm, Lindqvist and Seip, we consider different properties of dilations systems of a fixed function $\varphi \in L^2(0,1)$. More precisely, we study when the system $\{\varphi(nx)\}_n$ is a Bessel sequence, a…

This paper analyzes representations of continuous piecewise linear functions with infinite width, finite cost shallow neural networks using the rectified linear unit (ReLU) as an activation function. Through its integral representation, a…

It has been widely assumed that a neural network cannot be recovered from its outputs, as the network depends on its parameters in a highly nonlinear way. Here, we prove that in fact it is often possible to identify the architecture,…

We present a Riesz integral representation theory in which functions, operators and measures take values in uniform commutative monoids (a commutative monoid with a uniformity making the binary operation of the monoid uniformly continuous).…

This paper studies the approximation property of ReLU neural networks (NNs) to piecewise constant functions with unknown interfaces in bounded regions in $\mathbb{R}^d$. Under the assumption that the discontinuity interface $\Gamma$ may be…

We present a way to construct Parseval frames of piecewise constant functions for $L^2[0,1]$. The construction is similar to the generalized Walsh bases. It is based on iteration of operators that satisfy a Cuntz-type relation, but without…

Given an orthonormal basis $ {\mathcal V}= \{v_j\} _{j\in N}$ in a separable Hilbert space $H$ and a set of unit vectors $ {\mathcal B}=\{w_j\}_{j\in N}$, we consider the sets $ {\mathcal B}_N$ obtained by replacing the vectors $v_1, ...,\,…

We study two-layer neural networks whose domain and range are Banach spaces with separable preduals. In addition, we assume that the image space is equipped with a partial order, i.e. it is a Riesz space. As the nonlinearity we choose the…

The classic Riesz representation theorem characterizes all linear and increasing functionals on the space $C_{c}(X)$ of continuous compactly supported functions. A geometric version of this result, which characterizes all linear increasing…

We introduce a class of algebraic varieties naturally associated with ReLU neural networks, arising from the piecewise linear structure of their outputs across activation regions in input space, and the piecewise multilinear structure in…