Related papers: The attractors in sequence processing neural netwo…
A simple one-dimensional cellular automaton model with threshold dynamics is introduced. The cumulative distribution of the size of the relaxations is analytically computed and behaves as a power law with an exponent equal to -1. This…
Neural networks using transformer-based architectures have recently demonstrated great power and flexibility in modeling sequences of many types. One of the core components of transformer networks is the attention layer, which allows…
Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to…
We consider the storage properties of temporal patterns, i.e. cycles of finite lengths, in neural networks represented by (generally asymmetric) spin glasses defined on random graphs. Inspired by the observation that dynamics on sparse…
The standard approach to realize a quantum repeater relies upon probabilistic but heralded entangled state manipulations and the storage of quantum states while waiting for successful events. In the literature on this class of repeaters,…
We study associative memory neural networks of the Hodgkin-Huxley type of spiking neurons in which multiple periodic spatio-temporal patterns of spike timing are memorized as limit-cycle-type attractors. In encoding the spatio-temporal…
Sequences of neural activity arise in many brain areas, including cortex, hippocampus, and central pattern generator circuits that underlie rhythmic behaviors like locomotion. While network architectures supporting sequence generation vary…
Throughout the literature on Neural Cellular Automata (NCAs), it is often taken for granted that the systems learn attractors. This is shown through evolving the system for many timesteps and noting visual similarity to the goal state.…
Average-case analysis computes the complexity of an algorithm averaged over all possible inputs. Compared to worst-case analysis, it is more representative of the typical behavior of an algorithm, but remains largely unexplored in…
A class of models describing the flow of information within networks via routing processes is proposed and investigated, concentrating on the effects of memory traces on the global properties. The long-term flow of information is governed…
One of the limitations of transformer networks is the sequence length due to the quadratic nature of the attention matrix. Classical self attention uses the entire sequence length, however, the actual attention being used is sparse. Humans…
The brain is targeted for processing temporal sequence information. It remains largely unclear how the brain learns to store and retrieve sequence memories. Here, we study how recurrent networks of binary neurons learn sequence attractors…
As data-driven methods are deployed in real-world settings, the processes that generate the observed data will often react to the decisions of the learner. For example, a data source may have some incentive for the algorithm to provide a…
A version of ``preferential attachment'' random graphs, corresponding to linear ``weights'' with random ``edge additions,'' which generalizes some previously considered models, is studied. This graph model is embedded in a continuous-time…
We show how reaction coordinate path lengths affect the relaxation efficiency of a complex system. To this purpose, we consider the metric contributions to the transition rates. These metric contributions preserve informations about the…
We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the…
We investigate random walks on complex networks and derive an exact expression for the mean first passage time (MFPT) between two nodes. We introduce for each node the random walk centrality $C$, which is the ratio between its coordination…
In general, the power-law degree distribution has profound influence on various dynamical processes defined on scale-free networks. In this paper, we will show that power-law degree distribution alone does not suffice to characterize the…
The deterministic dynamics of randomly connected neural networks are studied, where a state of binary neurons evolves according to a discreet-time synchronous update rule. We give a theoretical support that the overlap of systems' states…
The first passage time (FPT) distribution for random walk in complex networks is calculated through an asymptotic analysis. For network with size $N$ and short relaxation time $\tau\ll N$, the computed mean first passage time (MFPT), which…