Related papers: Activation Saturation and Floquet Spectrum Collaps…
In this paper, we revise two commonly used saturated functions, the logistic sigmoid and the hyperbolic tangent (tanh). We point out that, besides the well-known non-zero centered property, slope of the activation function near the origin…
Smooth activation functions are ubiquitous in modern deep learning, yet their theoretical advantages over non-smooth counterparts remain poorly understood. In this work, we study both approximation and statistical properties of neural…
Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order…
We investigate the sample complexity of bounded two-layer neural networks using different activation functions. In particular, we consider the class $$ \mathcal{H} = \left\{\textbf{x}\mapsto \langle \textbf{v}, \sigma \circ W\textbf{b} +…
The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at…
Despite the huge number of neurons composing a brain network, ongoing activity of local cell assemblies composing cortical columns is intrinsically stochastic. Fluctuations in their instantaneous rate of spike firing $\nu(t)$ scale with the…
We study finite-element and deep feedforward neural network (DNN for short) expressivity rate bounds for solution sets of a model linear, second order singularly perturbed, elliptic two-point boundary value problem, in Sobolev norms on a…
We investigate the dynamical and spectral consequences of capacitance-network-mediated interactions in superconducting transmon arrays beyond effective nearest-neighbor descriptions. While weak coupling regimes are well captured by an…
Activation functions are critical to the performance of deep neural networks, particularly in domains such as functional near-infrared spectroscopy (fNIRS), where nonlinearity, low signal-to-noise ratio (SNR), and signal variability poses…
The article is devoted to the study of neural networks with one hidden layer and a modified activation function for solving physical problems. A rectified sigmoid activation function has been proposed to solve physical problems described by…
This article contributes to the current statistical theory of deep neural networks (DNNs). It was shown that DNNs are able to circumvent the so--called curse of dimensionality in case that suitable restrictions on the structure of the…
We prove deep neural network (DNN for short) expressivity rate bounds for solution sets of a model class of singularly perturbed, elliptic two-point boundary value problems, in Sobolev norms, on the bounded interval $(-1,1)$. We assume that…
Spatially uniform optical excitations can induce Floquet topological band structures within insulators which can develop similar or equal characteristics as are known from three-dimensional topological insulators. We derive in this article…
We study associative memory neural networks of the Hodgkin-Huxley type of spiking neurons in which multiple periodic spatio-temporal patterns of spike timing are memorized as limit-cycle-type attractors. In encoding the spatio-temporal…
Deep neural networks, as a powerful system to represent high dimensional complex functions, play a key role in deep learning. Convergence of deep neural networks is a fundamental issue in building the mathematical foundation for deep…
Activation functions govern the expressivity and stability of neural networks, yet existing comparisons remain largely heuristic. We propose a rigorous framework for their classification via a nine-dimensional integral signature…
We study a perturbed Floquet Hamiltonian $K+\beta V$ depending on a coupling constant $\beta$. The spectrum $\sigma(K)$ is assumed to be pure point and dense. We pick up an eigen-value, namely $0\in\sigma(K)$, and show the existence of a…
Activation in deep neural networks is fundamental to achieving non-linear mappings. Traditional studies mainly focus on finding fixed activations for a particular set of learning tasks or model architectures. The research on flexible…
Neural Ordinary Differential Equations (ODEs) have been successful in various applications due to their continuous nature and parameter-sharing efficiency. However, these unique characteristics also introduce challenges in training,…
We consider a perturbed Floquet Hamiltonian $-i\partial_t + H + \beta V(\omega t)$ in the Hilbert space $L^2([0,T],E,dt)$. Here $H$ is a self-adjoint operator in $E$ with a discrete spectrum obeying a growing gap condition, $V(t)$ is a…