Related papers: Polynomial, trigonometric, and tropical activation…

We provide new theoretical insights on why over-parametrization is effective in learning neural networks. For a $k$ hidden node shallow network with quadratic activation and $n$ training data points, we show as long as $ k \ge \sqrt{2n}$,…

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to…

The spectral decomposition for an explicit second-order differential operator $T$ is determined. The spectrum consists of a continuous part with multiplicity two, a continuous part with multiplicity one, and a finite discrete part with…

Matrix functions such as square root, inverse roots, and orthogonalization play a central role in preconditioned gradient methods for neural network training. This has motivated the development of iterative algorithms that avoid explicit…

We initiate the study of active learning polynomial threshold functions (PTFs). While traditional lower bounds imply that even univariate quadratics cannot be non-trivially actively learned, we show that allowing the learner basic access to…

The suitable basis functions for approximating periodic function are periodic, trigonometric functions. When the function is not periodic, a viable alternative is to consider polynomials as basis functions. In this paper we will point out…

Neural networks are generally built by interleaving (adaptable) linear layers with (fixed) nonlinear activation functions. To increase their flexibility, several authors have proposed methods for adapting the activation functions…

Orthogonal polynomials are of fundamental importance in many fields of mathematics and science, therefore the study of a particular family is always relevant. In this manuscript, we present a survey of some general results of the Hermite…

The application of graph theory to model the complex structure and function of the brain has shed new light on its organization and function, prompting the emergence of network neuroscience. Despite the tremendous progress that has been…

Even in recent neural network architectures such as Transformers and Extended LSTM (xLSTM), and traditional ones like Convolutional Neural Networks, Activation Functions are an integral part of nearly all neural networks. They enable more…

We study convolutional neural networks with monomial activation functions. Specifically, we prove that their parameterization map is regular and is an isomorphism almost everywhere, up to rescaling the filters. By leveraging on tools from…

We construct a generating functional for the exact evalutation of a coherent representation of spin network amplitudes. This generating functional is defined for arbitrary graphs and depends only on a pair of spinors for each edge. The…

We translate the usual class of partial/primitive recursive functions to a pointer recursion framework, accessing actual input values via a pointer reading unit-cost function. These pointer recursive functions classes are proven equivalent…

We introduce two q-analogues of the 2D-Hermite polynomials which are functions of two complex variables. We derive explicit formulas, orthogonality relations, raising and lowering operator relations, generating functions, and Rodrigues…

Deep neural networks such as GoogLeNet, ResNet, and BERT have achieved impressive performance in tasks such as image and text classification. To understand how such performance is achieved, we probe a trained deep neural network by studying…

Photometric transformations, such as brightness and contrast adjustment, can be applied to a face image repeatedly creating a set of near-duplicate images. Identifying the original image from a set of such near-duplicates and deducing the…

Deep neural networks have been the driving force behind the success in classification tasks, e.g., object and audio recognition. Impressive results and generalization have been achieved by a variety of recently proposed architectures, the…

An example of an activation function $\sigma$ is given such that networks with activations $\{\sigma, \lfloor\cdot\rfloor\}$, integer weights and a fixed architecture depending on $d$ approximate continuous functions on $[0,1]^d$. The range…

In this paper we introduce activation functions that move the entire computation of Convolutional Networks into the frequency domain, where they are actually Hadamard Networks. To achieve this result we employ the properties of Discrete…

In this paper two important classes of orthogonal polynomials in higher dimensions using the framework of Clifford analysis are considered, namely the Clifford-Hermite and the Clifford-Gegenbauer polynomials. For both classes an explicit…