Related papers: A Generalization Method of Partitioned Activation …
Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural…
In classical computational chemistry, the coupled-cluster ansatz is one of the most commonly used $ab~initio$ methods, which is critically limited by its non-unitary nature. The unitary modification as an ideal solution to the problem is,…
Complex-valued neural networks (CVNNs) have been shown to be powerful nonlinear approximators when the input data can be properly modeled in the complex domain. One of the major challenges in scaling up CVNNs in practice is the design of…
The paper presents partial-realization theory and realization algorithms for linear switched systems. Linear switched systems are a particular subclass of hybrid systems. We formulate a notion of a partial realization and we present…
Complex-valued neural networks are not a new concept, however, the use of real-valued models has often been favoured over complex-valued models due to difficulties in training and performance. When comparing real-valued versus…
In this paper, we investigate the relationship between deep neural networks (DNN) with rectified linear unit (ReLU) function as the activation function and continuous piecewise linear (CPWL) functions, especially CPWL functions from the…
Real-world analog systems intrinsically suffer from noise that can impede model convergence and accuracy on a variety of deep learning models. We demonstrate that differentiable activations like GELU and SiLU enable robust propagation of…
Different activation functions work best for different deep learning models. To exploit this, we leverage recent advancements in gradient-based search techniques for neural architectures to efficiently identify high-performing activation…
Explicit expressions for multimatrix models with complex and unitary matrices allows to couple these models with well-known unitary, orthogonsl and sympletic ensembles. We consider examples of such mixed ensembles which are solvable in the…
Recent work by Craig, van Ittersum, and Ono constructs explicit expressions in the partition functions of MacMahon that detect the prime numbers. Furthermore, they define generalizations, the MacMahonesque functions, and prove there are…
Partition of unity methods (PUMs) on graphs are simple and highly adaptive auxiliary tools for graph signal processing. Based on a greedy-type metric clustering and augmentation scheme, we show how a partition of unity can be generated in…
Designing effective neural networks requires tuning architectural elements. This study integrates fractional calculus into neural networks by introducing fractional order derivatives (FDO) as tunable parameters in activation functions,…
Complex polynomial optimization has recently gained more and more attention in both theory and practice. In this paper, we study the optimization of a real-valued general conjugate complex form over various popular constraint sets including…
Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training. To encourage more systematic numerical extrapolation, we…
The aim of this note is to provoke discussion concerning arithmetic properties of function $p_{d}(n)$ counting partitions of an positive integer $n$ into $d$-th powers, where $d\geq 2$. Besides results concerning the asymptotic behavior of…
Recently, much attention has been devoted to finding highly efficient and powerful activation functions for CNN layers. Because activation functions inject different nonlinearities between layers that affect performance, varying them is one…
Dynamics of a charged particle in the canonical coordinates is a Hamiltonian system, and the well-known symplectic algorithm has been regarded as the de facto method for numerical integration of Hamiltonian systems due to its long-term…
Fractional calculus has been used to describe physical systems with complexity. Here, we show that a fractional calculus approach can restore or include complexity in any physical systems that can be described by partial differential…
We propose an approach for approximating the partition function which is based on two steps: (1) computing the partition function of a simplified model which is obtained by deleting model edges, and (2) rectifying the result by applying an…
We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, we choose between SILU or RELU depending on a Bernoulli draw.…