Related papers: Associative Memory by Recurrent Neural Networks wi…
We show that discrete synaptic weights can be efficiently used for learning in large scale neural systems, and lead to unanticipated computational performance. We focus on the representative case of learning random patterns with binary…
This work focuses on the training dynamics of one associative memory module storing outer products of token embeddings. We reduce this problem to the study of a system of particles, which interact according to properties of the data…
Time-delayed oscillator networks underlie diverse biological and physical systems, yet standard first-order phase reductions fail to capture their high-dimensional collective dynamics. In this Letter, we develop a universal second-order…
We study extremely diluted spin models of neural networks in which the connectivity evolves in time, although adiabatically slowly compared to the neurons, according to stochastic equations which on average aim to reduce frustration. The…
Recurrent neural networks (RNNs) and self-attention are both widely used sequence-mixing layers that maintain an internal memory. However, this memory is constructed using two orthogonal mechanisms: RNNs compress the entire past into a…
A problem related to the development of an algorithm designed to find an architecture of artificial neural network used for black-box modelling of dynamic systems with time delays has been addressed in this paper. The proposed algorithm is…
We solve the dynamics of Hopfield-type neural networks which store sequences of patterns, close to saturation. The asymmetry of the interaction matrix in such models leads to violation of detailed balance, ruling out an equilibrium…
Discrete dynamics arise naturally in systems with broken temporal translation symmetry and are typically described by first-order recurrence relations representing classical or quantum Markov chains. When memory effects induced by hidden…
We present a Hopfield-like autoassociative network for memories representing examples of concepts. Each memory is encoded by two activity patterns with complementary properties. The first is dense and correlated across examples within…
For large fully connected neuron networks, we study the dynamics of homogenous assemblies of interacting neurons described by time elapsed models, indicating how the time elapsed since the last discharge construct the probability density of…
The memory demands of large-scale deep neural networks (DNNs) require synaptic weight values to be stored and updated in off-chip memory like dynamic random-access memory, which reduces energy efficiency and increases training time.…
We design a linear chain trick algorithm for dynamical systems for which we have oscillatory time histories in the distributed time delay. We make use of this algorithmic framework to analyse memory effects in disease evolution in a…
Sequence models face a fundamental tradeoff between memory capacity and computational efficiency. Transformers achieve expressive context modeling at quadratic cost, while linear attention and state-space models run in linear time by…
This paper considers optimization of power and delay in a time-varying wireless link using rateless codes. The link serves a sequence of variable-length packets. Each packet is coded and transmitted over multiple slots. Channel conditions…
Large Language Models (LLMs) struggle to handle long input sequences due to high memory and runtime costs. Memory-augmented models have emerged as a promising solution to this problem, but current methods are hindered by limited memory…
We introduce and study the problem of Online Continual Compression, where one attempts to simultaneously learn to compress and store a representative dataset from a non i.i.d data stream, while only observing each sample once. A naive…
Statistical learning in high-dimensional spaces is challenging without a strong underlying data structure. Recent advances with foundational models suggest that text and image data contain such hidden structures, which help mitigate the…
In this paper, we consider a model of impulsive recurrent neural networks with piecewise constant delay. The dynamics are presented by differential equations with discontinuities such as impulses at fixed moments and piecewise constant…
Time-delay systems are an important class of dynamical systems that provide a solid mathematical framework to deal with many application domains of interest. In this paper we focus on nonlinear control systems with unknown and time-varying…
We analyze the macroscopic behavior of multi-populations randomly connected neural networks with interaction delays. Similar to cases occurring in spin glasses, we show that the sequences of empirical measures satisfy a large deviation…