Related papers: Long Sequence Hopfield Memory
Associative memory models, in theoretical neuro- and computer sciences, can generally store a sublinear number of memories. We show that using quantum annealing for recall tasks endows associative memory models with exponential storage…
Traditional studies of memory for meaningful narratives focus on specific stories and their semantic structures but do not address common quantitative features of recall across different narratives. We introduce a statistical ensemble of…
The Hopfield network model and its generalizations were introduced as a model of associative, or content-addressable, memory. They were widely investigated both as an unsupervised learning method in artificial intelligence and as a model of…
It is known that Recurrent Neural Networks (RNNs) can remember, in their hidden layers, part of the semantic information expressed by a sequence (e.g., a sentence) that is being processed. Different types of recurrent units have been…
Memories in the brain are separated in two categories: short-term and long-term memories. Long-term memories remain for a lifetime, while short-term ones exist from a few milliseconds to a few minutes. Within short-term memory studies,…
The long-term memory of most connectionist systems lies entirely in the weights of the system. Since the number of weights is typically fixed, this bounds the total amount of knowledge that can be learned and stored. Though this is not…
Recurrent neural network is a powerful model that learns temporal patterns in sequential data. For a long time, it was believed that recurrent networks are difficult to train using simple optimizers, such as stochastic gradient descent, due…
The notion of memory capacity, originally introduced for echo state and linear networks with independent inputs, is generalized to nonlinear recurrent networks with stationary but dependent inputs. The presence of dependence in the inputs…
Transformer architectures have achieved state-of-the-art results on a variety of sequence modeling tasks. However, their attention mechanism comes with a quadratic complexity in sequence lengths, making the computational overhead…
In this paper we address the question of how to render sequence-level networks better at handling structured input. We propose a machine reading simulator which processes text incrementally from left to right and performs shallow reasoning…
Intracortical brain-machine interfaces require decoders that adapt continuously to neural signal instability while operating within strict memory budgets. We introduce a dual-timescale Hebbian accumulator learning rule for spiking neural…
Among the performance-enhancing procedures for Hopfield-type networks that implement associative memory, Hebbian Unlearning (or dreaming) strikes for its simplicity and its clear biological interpretation. Yet, it does not easily lend…
Sequence models assign probabilities to variable-length sequences such as natural language texts. The ability of sequence models to capture temporal dependence can be characterized by the temporal scaling of correlation and mutual…
Hopfield models, originally developed to study memory retrieval in neural networks, have become versatile tools for modeling diverse biological systems in which function emerges from collective dynamics. In this review, we provide a…
We show how a Hopfield network with modifiable recurrent connections undergoing slow Hebbian learning can extract the underlying geometry of an input space. First, we use a slow/fast analysis to derive an averaged system whose dynamics…
Continual learning is the ability to acquire new knowledge without forgetting the previously learned one, assuming no further access to past training data. Neural network approximators trained with gradient descent are known to fail in this…
A longstanding challenge for the Machine Learning community is the one of developing models that are capable of processing and learning from very long sequences of data. The outstanding results of Transformers-based networks (e.g., Large…
Processing sequential inputs is a fundamental brain function, underlying tasks such as sensory perception, language, and motor control. A challenge in sequence processing is to represent not only the order of events, but also their precise…
The standard model of memory consolidation foresees that memories are initially recorded in the hippocampus, while features that capture higher-level generalisations of data are created in the cortex, where they are stored for a possibly…
Effectively learning from sequential data is a longstanding goal of Artificial Intelligence, especially in the case of long sequences. From the dawn of Machine Learning, several researchers have pursued algorithms and architectures capable…