Related papers: The three-state layered neural network with finite…
The loss surface of deep neural networks has recently attracted interest in the optimization and machine learning communities as a prime example of high-dimensional non-convex problem. Some insights were recently gained using spin glass…
Kinetics of a balanced network of neurons with a sparse grid of synaptic links is well representable by the stochastic dynamics of a generic neuron subject to an effective shot noise. The rate of delta-pulses of the noise is determined…
Recent empirical studies have identified fixed point iteration phenomena in deep neural networks, where the hidden state tends to stabilize after several layers, showing minimal change in subsequent layers. This observation has spurred the…
Recurrent neural network architectures can have useful computational properties, with complex temporal dynamics and input-sensitive attractor states. However, evaluation of recurrent dynamic architectures requires solution of systems of…
We study how different output layer parameterizations of a deep neural network affects learning and forgetting in continual learning settings. The following three effects can cause catastrophic forgetting in the output layer: (1) weights…
The parallel dynamics of the asymmetric extremely diluted Ashkin-Teller neural network is studied using signal-to-noise analysis techniques. Evolution equations for the order parameters are derived, both at zero and finite temperature. The…
The objective of this paper is to enhance the optimization process for neural networks by developing a dynamic learning rate algorithm that effectively integrates exponential decay and advanced anti-overfitting strategies. Our primary…
It has been argued in the past that high-dimensional neural networks do not exhibit local minima capable of trapping an optimisation algorithm. However, the relationship between loss surface modality and the neural architecture parameters,…
Here we numerically study a model of excitable media, namely, a network with occasionally quiet nodes and connection weights that vary with activity on a short-time scale. Even in the absence of stimuli, this exhibits unstable dynamics,…
The full dynamics of a synchronous recurrent neural network model with Ising binary units and a Hebbian learning rule with a finite self-interaction is studied in order to determine the stability to synaptic and stochastic noise of…
Deep networks are commonly used to model dynamical systems, predicting how the state of a system will evolve over time (either autonomously or in response to control inputs). Despite the predictive power of these systems, it has been…
We perform network analysis of a system described by the master equation to estimate the lower bound of the steady-state current noise, starting from the level 2.5 large deviation function and using the graph theory approach. When the…
Substantial work indicates that the dynamics of neural networks (NNs) is closely related to their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs with infinite width (Luo et al., 2021), we make a step…
We consider a neural network with adapting synapses whose dynamics can be analitically computed. The model is made of $N$ neurons and each of them is connected to $K$ input neurons chosen at random in the network. The synapses are…
We develop a general theory of synaptic neural balance and how it can emerge or be enforced in neural networks. For a given regularizer, a neuron is said to be in balance if the total cost of its input weights is equal to the total cost of…
Diagonal linear networks (DLNs) are a tractable model that captures several nontrivial behaviors in neural network training, such as initialization-dependent solutions and incremental learning. These phenomena are typically studied in…
We disclose the generality of the intrinsic mechanisms underlying multistability in reciprocally inhibitory 3-cell circuits composed of simplified, low-dimensional models of oscillatory neurons, as opposed to those of a detailed Hodgkin-…
We study deterministic continuous-time lossy dynamical flow networks with constant exogenous demands, fixed routing, and finite flow and buffer capacities. In the considered model, when the total net flow in a cell ---consisting of the…
Deep unrolling, or unfolding, is an emerging learning-to-optimize method that unrolls a truncated iterative algorithm in the layers of a trainable neural network. However, the convergence guarantees and generalizability of the unrolled…
With the rise of big data analytics, multi-layer neural networks have surfaced as one of the most powerful machine learning methods. However, their theoretical mathematical properties are still not fully understood. Training a neural…