相关论文: Hardware Realization of Nonlinear Activation Funct…

When training early-stage deep neural networks (DNNs), generating intermediate features via convolution or linear layers occupied most of the execution time. Accordingly, extensive research has been done to reduce the computational burden…

Lithium-ion battery packs demand effective active equalization systems to enhance their usable capacity and lifetime. Despite numerous topologies and control schemes proposed in the literature, conducting quantitative analyses,…

Previous works proved that the combination of the linear neuron network with nonlinear activation functions (e.g. ReLu) can achieve nonlinear function approximation. However, simply widening or deepening the network structure will introduce…

In this paper, we study neural networks from the point of view of nonsmooth optimisation, namely, quasidifferential calculus. We restrict ourselves to the case of uniform approximation by a neural network without hidden layers, the…

In this paper, we extend an available neural network verification technique to support a wider class of piece-wise linear activation functions. Furthermore, we extend the algorithms, which provide in their original form exact respectively…

The scope of research in the domain of activation functions remains limited and centered around improving the ease of optimization or generalization quality of neural networks (NNs). However, to develop a deeper understanding of deep…

Optical approaches have made great strides towards the goal of high-speed, energy-efficient computing necessary for modern deep learning and AI applications. Read-in and read-out of data, however, limit the overall performance of existing…

It is shown that the port behavior of a resistor-diode network corresponds to the solution of a ReLU monotone operator equilibrium network (a neural network in the limit of infinite depth), giving a parsimonious construction of a neural…

Recent seminal work at the intersection of deep neural networks practice and random matrix theory has linked the convergence speed and robustness of these networks with the combination of random weight initialization and nonlinear…

Bidirectional recurrent neural networks (bi-RNNs), in particular, bidirectional long short term memory (bi-LSTM), bidirectional gated recurrent unit, and convolutional bi-LSTM models have recently attracted attention for nonlinearity…

In this paper, we develop a wavelet-based theoretical framework for analyzing the universal approximation capabilities of neural networks over a wide range of activation functions. Leveraging wavelet frame theory on the spaces of…

This paper introduces a $H_\infty$-like methodology of coherent filtering for equalization of passive linear quantum systems to help mitigate degrading effects of quantum communication channels. For such systems, which include a wide range…

Deep neural networks (DNNs) have substantial computational requirements, which greatly limit their performance in resource-constrained environments. Recently, there are increasing efforts on optical neural networks and optical computing…

Photonic neural networks have demonstrated their potential over the past decades, but have not yet reached the full extent of their capabilities. One reason for this lies in an essential component - the nonlinear activation function, which…

This paper investigates the approximation properties of shallow neural networks with activation functions that are powers of exponential functions. It focuses on the dependence of the approximation rate on the dimension and the smoothness…

This paper presents new structure and adaptation criterion for equalization of two-dimensional magnetic recording channels, as opposed to typical linear equalizer with minimum mean square error (MMSE) as adaptation criterion. To compensate…

This paper presents a novel unifying framework of bilinear LSTMs that can represent and utilize the nonlinear interaction of the input features present in sequence datasets for achieving superior performance over a linear LSTM and yet not…

Integrated optical devices that can realize a threshold filtration of signals are in demand in photonics. In particular, they play a key role in neuromorphic chips, acting as optical neurons. A list of requirements exist for thresholders to…

The conjugation of multiple spatial light modulators (SLMs) enables the construction of optical diffractive neural networks (DNNs). To accelerate training, which is limited by the low refresh rate of SLMs, spatial multiplexing of the input…

Brain-Like Stochastic Search (BLiSS) refers to this task: given a family of utility functions U(u,A), where u is a vector of parameters or task descriptors, maximize or minimize U with respect to u, using networks (Option Nets) which input…