Related papers: Transient dynamics of associative memory models
Hopfield models, originally developed to study memory retrieval in neural networks, have become versatile tools for modeling diverse biological systems in which function emerges from collective dynamics. In this review, we provide a…
Recent interest has developed around the problem of dynamic compressed sensing, or the recovery of time-varying, sparse signals from limited observations. In this paper, we study how the dynamics of recurrent networks, formulated as general…
Following a stimulus, the neural response typically strongly varies in time and across neurons before settling to a steady-state. While classical population coding theory disregards the temporal dimension, recent works have argued that…
Humans learn and form memories in stochastic environments. Auto-associative memory systems model these processes by storing patterns and later recovering them from corrupted versions. Here, memories are learned by associating each pattern…
The task of a neural associative memory is to retrieve a set of previously memorized patterns from their noisy versions using a network of neurons. An ideal network should have the ability to 1) learn a set of patterns as they arrive, 2)…
We study a discrete-time interacting particle system with continuous state space which is motivated by a mathematical model for turnover through branching in actin filament networks. It gives rise to transient clusters reminiscent of actin…
Attractors of dynamical systems may be networks in phase space that can be heteroclinic (where there are dynamical connections between simple invariant sets) or excitable (where a perturbation threshold needs to be crossed to a dynamical…
Catastrophic forgetting and capacity saturation are the central challenges of any parametric lifelong learning system. In this work, we study these challenges in the context of sequential supervised learning with an emphasis on recurrent…
We analyze how the transient dynamics of large dynamical systems in the vicinity of a stationary point, modeled by a set of randomly coupled linear differential equations, depends on the network topology. We characterize the transient…
Retentive Network (RetNet) represents a significant advancement in neural network architecture, offering an efficient alternative to the Transformer. While Transformers rely on self-attention to model dependencies, they suffer from high…
Generalized Hopfield models with higher-order or exponential interaction terms are known to have substantially larger storage capacities than the classical quadratic model. On the other hand, associative memories for sparse patterns, such…
We discuss adiabatic spectra and dynamics of the quantum, i.e. transverse field, Hopfield model with dilute memories (the number of stored patterns $p < log_2 N$, where $N$ is the number of qubits). At some critical transverse field the…
Complex systems of many interacting components exhibit patterns of recurrence and emergent behaviors in their time evolution that can be understood from a new perspective of physics of information dynamics, modeled after one such system,…
We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfield networks. Using a Bayesian framework, we show…
The training of neural networks is a complex, high-dimensional, non-convex and noisy optimization problem whose theoretical understanding is interesting both from an applicative perspective and for fundamental reasons. A core challenge is…
The comprehension of the mechanisms at the basis of the functioning of complexly interconnected networks represents one of the main goals of neuroscience. In this work, we investigate how the structure of recurrent connectivity influences…
As shown recently (arXiv:0801.3056), several types of neuronal complex networks involving non-linear integration-and-fire dynamics exhibit an abrupt activation along their transient regime. Interestingly, such an avalanche of activation has…
The potential for associative recall of diluted neuronal networks is investigated with respect to several biologically relevant configurations, more specifically the position of the cells along the input space and the spatial distribution…
We explore statistical characteristics of avalanches associated with the dynamics of a complex-network model, where two modules corresponding to sensorial and symbolic memories interact, representing unconscious and conscious mental…
Neural dynamics of energy-based models are governed by energy minimization and the patterns stored in the network are retrieved when the system reaches equilibrium. However, when the system is driven by time-varying external input, the…