Related papers: Statistical Neurodynamics for sequence processing …
An exact solution of the transient dynamics for a sequential associative memory model is discussed through both the path-integral method and the statistical neurodynamics. Although the path-integral method has the ability to give an exact…
Analyzing neural network dynamics via stochastic gradient descent (SGD) is crucial to building theoretical foundations for deep learning. Previous work has analyzed structured inputs within the \textit{hidden manifold model}, often under…
Node-perturbation learning is a type of statistical gradient descent algorithm that can be applied to problems where the objective function is not explicitly formulated, including reinforcement learning. It estimates the gradient of an…
The theoretical basis of neuronal coding, associated with short term degradation in synaptic transmission, is a matter of debate in the literature. In fact, electrophysiological signals are commonly characterized as inversely proportional…
The stochastic gradient descent (SGD) algorithm is the algorithm we use to train neural networks. However, it remains poorly understood how the SGD navigates the highly nonlinear and degenerate loss landscape of a neural network. In this…
Estimating parameters of drift and diffusion coefficients for multidimensional stochastic delay equations with small noise are considered. The delay structure is written as an integral form with respect to a delay measure. Our contrast…
Strong, correlated noise in recurrent neural circuits often passes through nonlinear transfer functions, complicating dynamical mean-field analyses of complex phenomena such as transients and bifurcations. We introduce a method that…
We propose stochastic, non-parametric activation functions that are fully learnable and individual to each neuron. Complexity and the risk of overfitting are controlled by placing a Gaussian process prior over these functions. The result is…
Noise is an inherent part of neuronal dynamics, and thus of the brain. It can be observed in neuronal activity at different spatiotemporal scales, including in neuronal membrane potentials, local field potentials, electroencephalography,…
Recent work has shown that training wide neural networks with gradient descent is formally equivalent to computing the mean of the posterior distribution in a Gaussian Process (GP) with the Neural Tangent Kernel (NTK) as the prior…
A theoretical approach for characterising the influence of asymmetry of noise distribution on the escape rate of a multi-stable system is presented. This was carried out via the estimation of an action, which is defined as an exponential…
A complete first and second order statistical characterization of noise in SENSE reconstructed data is proposed. SENSE acquisitions have usually been modeled as Rician distributed, since the data reconstruction takes place into the spatial…
The dynamical organization in the presence of noise of a Boolean neural network with random connections is analyzed. For low levels of noise, the system reaches a stationary state in which the majority of its elements acquire the same…
Recent experiments have shown that the spontaneous activity of young dissociated neuronal cultures can be described as a process of highly inhomogeneous nucleation and front propagation due to the localization of noise activity, i.e., noise…
The fluctuating dynamics of a network about its stable, noise-free steady state are theoretically investigated. Various causes of non-equilibrium dynamics are identified in terms of the properties and symmetry of the network connections and…
A lecture notes style review of the non-equilibrium statistical mechanics of recurrent neural networks with discrete and continuous neurons (e.g. Ising, graded-response, coupled-oscillators). To be published in the Handbook of Biological…
We derive a analytic evolution equation for overlap parameters including the effect of degree distribution on the transient dynamics of sequence processing neural networks. In the special case of globally coupled networks, the precisely…
In this paper, we characterize the noise of stochastic gradients and analyze the noise-induced dynamics during training deep neural networks by gradient-based optimizers. Specifically, we firstly show that the stochastic gradient noise…
Neural signals are characterized by rich temporal and spatiotemporal dynamics that reflect the organization of cortical networks. Theoretical research has shown how neural networks can operate at different dynamic ranges that correspond to…
Recurrent networks of dynamic elements frequently exhibit emergent collective oscillations, which can display substantial regularity even when the individual elements are considerably noisy. How noise-induced dynamics at the local level…