Related papers: Transient dynamics of associative memory models
We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks. Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of…
Despite the striking successes of deep neural networks trained with gradient-based optimization, these methods differ fundamentally from their biological counterparts. This gap raises key questions about how nature achieves robust,…
Energy-based probabilistic models learned by maximizing the likelihood of the data are limited by the intractability of the partition function. A widely used workaround is to maximize the pseudo-likelihood, which replaces the global…
Quantum neural networks form one pillar of the emergent field of quantum machine learning. Here, quantum generalisations of classical networks realizing associative memories - capable of retrieving patterns, or memories, from corrupted…
Many real-world networks are directed, sparse and hierarchical, with a mixture of feed-forward and feedback connections with respect to the hierarchy. Moreover, a small number of 'master' nodes are often able to drive the whole system. We…
We test four fast mean field type algorithms on Hopfield networks as an inverse Ising problem. The equilibrium behavior of Hopfield networks is simulated through Glauber dynamics. In the low temperature regime, the simulated annealing…
Real systems are usually composed by units or nodes whose activity can be interrupted and restored intermittently due to complex interactions not only with the environment, but also with the same system. Majdand\v{z}i\'c $et\;al.$ [Nature…
We characterize the pre-softmax attention matrix $\mathbf{QK^\top}$ in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric…
The standard Hopfield model for associative neural networks accounts for biological Hebbian learning and acts as the harmonic oscillator for pattern recognition, however its maximal storage capacity is $\alpha \sim 0.14$, far from the…
We show that the dynamics of simple disordered models, like the directed Trap Model and the Random Energy Model, takes place at a coexistence point between active and inactive dynamical phases. We relate the presence of a dynamic phase…
Artificial Neural Networks has struggled to devise a way to incorporate working memory into neural networks. While the ``long term'' memory can be seen as the learned weights, the working memory consists likely more of dynamical activity,…
We consider statistical-mechanical models for spin systems built on hierarchical structures, which provide a simple example of non-mean-field framework. We show that the coupling decay with spin distance can give rise to peculiar features…
Recent research has established a connection between modern Hopfield networks (HNs) and transformer attention heads, with guarantees of exponential storage capacity. However, these models still face challenges scaling storage efficiently.…
This paper is a step towards a systematic theory of the transitivity (clustering) phenomenon in random networks. A static framework is used, with adjacency matrix playing the role of the dynamical variable. Hence, our model is a matrix…
Studies of the dynamics of nonlinear recurrent neural networks often assume independent and identically distributed couplings, but large-scale connectomics data indicate that biological neural circuits exhibit markedly different…
In networked systems, the interplay between the dynamics of individual subsystems and their network interactions has been found to generate multistability in various contexts. Despite its ubiquity, the specific mechanisms and ingredients…
We introduce a Hopfield-type associative memory in which effective connectivity is multiplicatively modulated by astrocytic gains evolving under an entropy-regularized replicator equation. The coupled neuron-astrocyte dynamics admit a…
Developing accurate and computationally inexpensive models for the dynamics of open-quantum systems is critical in designing new qubit platforms by first understanding their mechanisms of decoherence and dephasing. Current models based on…
Despite the significance of short-term memory in cognitive function, the process of encoding and sustaining the input information in neural activity dynamics remains elusive. Herein, we unveiled the significance of transient neural dynamics…
We consider a class of open quantum many-body systems that evolves in a Markovian fashion, the dynamical generator being in GKS-Lindblad form. Here, the Hamiltonian contribution is characterized by an all-to-all coupling, and the…