Related papers: Sufficient Conditions for Persistency of Excitatio…
This paper presents sufficient conditions for the stability and $\ell_2$-gain performance of recurrent neural networks (RNNs) with ReLU activation functions. These conditions are derived by combining Lyapunov/dissipativity theory with…
When an online learning algorithm is used to estimate the unknown parameters of a model, the signals interacting with the parameter estimates should not decay too quickly for the optimal values to be discovered correctly. This requirement…
The persistence of excitation (PE) condition is sufficient to ensure parameter convergence in adaptive estimation problems. Recent results on adaptive estimation in reproducing kernel Hilbert spaces (RKHS) introduce PE conditions for RKHS.…
ReLU is a widely used activation function in deep neural networks. This paper explores the stability properties of the ReLU map. For any weight matrix $\boldsymbol{A} \in \mathbb{R}^{m \times n}$ and bias vector $\boldsymbol{b} \in…
To enhance the nonlinearity of neural networks and increase their mapping abilities between the inputs and response variables, activation functions play a crucial role to model more complex relationships and patterns in the data. In this…
This Letter provides necessary and sufficient conditions on the excitation and measurement pattern (EMP) that guarantee identifiability of a dynamical network that has the structure of a loop. The conditions are extremely simple in their…
We study the reachability problem for systems implemented as feed-forward neural networks whose activation function is implemented via ReLU functions. We draw a correspondence between establishing whether some arbitrary output can ever be…
Selecting the most suitable activation function is a critical factor in the effectiveness of deep learning models, as it influences their learning capacity, stability, and computational efficiency. In recent years, the Gaussian Error Linear…
We study the problem of learning one-hidden-layer neural networks with Rectified Linear Unit (ReLU) activation function, where the inputs are sampled from standard Gaussian distribution and the outputs are generated from a noisy teacher…
We analyze a simple one-hidden-layer neural network with ReLU activation functions and fixed biases, with one-dimensional input and output. We study both continuous and discrete versions of the model, and we rigorously prove the convergence…
In the context of data-driven control, persistence of excitation (PE) of an input sequence is defined in terms of a rank condition on the Hankel matrix of the input data. For nonlinear systems, recent results employed rank conditions…
The paper formulates the concept of persistence of excitation for discrete-time linear switched systems, and provides sufficient conditions for an input signal to be persistently exciting. Persistence of excitation is formulated as a…
In this paper, we explore the concept of adding learn-able slope and mean shift parameters to an activation function to improve the total response region. The characteristics of an activation function depend highly on the value of…
Current research suggests that the key factors in designing neural network architectures involve choosing number of filters for every convolution layer, number of hidden neurons for every fully connected layer, dropout and pruning. The…
Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the…
The nonlinearity of activation functions used in deep learning models are crucial for the success of predictive models. There are several commonly used simple nonlinear functions, including Rectified Linear Unit (ReLU) and Leaky-ReLU…
In this paper, we consider the problem of estimating parameters of a linear regression model. Using a hybrid systems framework, a hybrid algorithm is proposed allowing the estimate to converge to the exact value of the unknown parameters in…
Activation functions are non-linearities in neural networks that allow them to learn complex mapping between inputs and outputs. Typical choices for activation functions are ReLU, Tanh, Sigmoid etc., where the choice generally depends on…
The training process of neural networks usually optimize weights and bias parameters of linear transformations, while nonlinear activation functions are pre-specified and fixed. This work develops a systematic approach to constructing…
The success of deep networks has been attributed in part to their expressivity: per parameter, deep networks can approximate a richer class of functions than shallow networks. In ReLU networks, the number of activation patterns is one…