Related papers: Elementary superexpressive activations
To a subshift over a finite alphabet, one can naturally associate an infinite family of finite graphs, called its Rauzy graphs. We show that for a subshift of subexponential complexity the Rauzy graphs converge to the line $\mathbf{Z}$ in…
We derive fundamental lower bounds on the connectivity and the memory requirements of deep neural networks guaranteeing uniform approximation rates for arbitrary function classes in $L^2(\mathbb R^d)$. In other words, we establish a…
In 1989 George Cybenko proved in a landmark paper that wide shallow neural networks can approximate arbitrary continuous functions on a compact set. This universal approximation theorem sparked a lot of follow-up research. Shen, Yang and…
Probabilistic circuits (PCs) are a family of generative models which allows for the computation of exact likelihoods and marginals of its probability distributions. PCs are both expressive and tractable, and serve as popular choices for…
Staged tree models are a discrete generalization of Bayesian networks. We show that these form curved exponential families and derive their natural parameters, sufficient statistic, and cumulant-generating function as functions of their…
We propose a family of models to study the evolution of ties in a network of interacting agents by reinforcement and penalization of their connections according to certain local laws of interaction. The family of stochastic dynamical…
Any function can be constructed using a hierarchy of simpler functions through compositions. Such a hierarchy can be characterized by a binary rooted tree. Each node of this tree is associated with a function which takes as inputs two…
Many biological decision-making processes can be viewed as performing a classification task over a set of inputs, using various chemical and physical processes as "biological hardware." In this context, it is important to understand the…
Relu Fully Connected Networks are ubiquitous but uninterpretable because they fit piecewise linear functions emerging from multi-layered structures and complex interactions of model weights. This paper takes a novel approach to piecewise…
Classical results in neural network approximation theory show how arbitrary continuous functions can be approximated by networks with a single hidden layer, under mild assumptions on the activation function. However, the classical theory…
Biological networks are customarily described as structurally robust. This means that they often function extremely well under large forms of perturbations affecting both the concentrations and the kinetic parameters. In order to explain…
We study two different types of (maximal) almost disjoint families: very mad families and (maximal) cofinitary groups. For the very mad families we prove the basic existence results. We prove that MA implies there exist many pairwise…
We study a new class of functions that arise naturally in quaternionic analysis, we call them "quasi regular functions". Like the well-known quaternionic regular functions, these functions provide representations of the quaternionic…
We show that a free action $G \curvearrowright X$ is almost finite if its restriction to some infinite normal subgroup of $G$ is almost finite. Consider the class of groups which contains all infinite groups of locally subexponential growth…
Many biological learning systems such as the mushroom body, hippocampus, and cerebellum are built from sparsely connected networks of neurons. For a new understanding of such networks, we study the function spaces induced by sparse random…
We provide an overview of several non-linear activation functions in a neural network architecture that have proven successful in many machine learning applications. We conduct an empirical analysis on the effectiveness of using these…
Many deep Convolutional Neural Networks (CNN) make incorrect predictions on adversarial samples obtained by imperceptible perturbations of clean samples. We hypothesize that this is caused by a failure to suppress unusual signals within…
In this article, we prove approximation theorems in classes of deep and shallow neural networks with analytic activation functions by elementary arguments. We prove for both real and complex networks with non-polynomial activation that the…
We study the approximation of functions by tensor networks (TNs). We show that Lebesgue $L^p$-spaces in one dimension can be identified with tensor product spaces of arbitrary order through tensorization. We use this tensor product…
In Radhakrishnan et al. [2020], the authors empirically show that autoencoders trained with usual SGD methods shape out basins of attraction around their training data. We consider network functions of width not exceeding the input…