Related papers: Forgetting Memories and their Attractiveness

We study various models of associative memories with sparse information, i.e. a pattern to be stored is a random string of $0$s and $1$s with about $\log N$ $1$s, only. We compare different synaptic weights, architectures and retrieval…

Human memory is fleeting. As words are processed, the exact wordforms that make up incoming sentences are rapidly lost. Cognitive scientists have long believed that this limitation of memory may, paradoxically, help in learning language -…

Nowadays, machine learning models, especially neural networks, become prevalent in many real-world applications.These models are trained based on a one-way trip from user data: as long as users contribute their data, there is no way to…

It has been observed that neural networks perform poorly when the data or tasks are presented sequentially. Unlike humans, neural networks suffer greatly from catastrophic forgetting, making it impossible to perform life-long learning. To…

I review and expand the model of quantum associative memory that I have recently proposed. In this model binary patterns of n bits are stored in the quantum superposition of the appropriate subset of the computational basis of n qbits.…

Modern language models are powerful, but typically static after deployment. A major obstacle to building models that continually learn over time is catastrophic forgetting, where updating on new data erases previously acquired capabilities.…

The main problem about replacing LTP as a memory mechanism has been to find other highly abstract, easily understandable principles for induced plasticity. In this paper we attempt to lay out such a basic mechanism, namely intrinsic…

Artificial neural networks thrive in solving the classification problem for a particular rigid task, acquiring knowledge through generalized learning behaviour from a distinct training phase. The resulting network resembles a static entity…

Understanding memorisation in language models has practical and societal implications, e.g., studying models' training dynamics or preventing copyright infringements. Prior work defines memorisation as the causal effect of training with an…

Catastrophic forgetting remains a formidable obstacle to building an omniscient model in large language models (LLMs). Despite the pioneering research on task-level forgetting in LLM fine-tuning, there is scant focus on forgetting during…

The idea of using metaplastic synapses to incorporate the separate storage of long- and short-term memories via an array of hidden states was put forward in the cascade model of Fusi et al. In this paper, we devise and investigate two…

In the present era of deep learning, continual learning research is mainly focused on mitigating forgetting when training a neural network with stochastic gradient descent on a non-stationary stream of data. On the other hand, in the more…

Memory plays a vital role in the temporal evolution of interactions of complex systems. To address the impact of memory on the temporal pattern of networks, we propose a simple preferential connection model, in which nodes have a…

Out-of-equilibrium disordered systems may form memories of external driving in a remarkable fashion. The system "remembers" multiple values from a series of training inputs yet "forgets" nearly all of them at long times despite the inputs…

In this lecture I will present some models of neural networks that have been developed in the recent years. The aim is to construct neural networks which work as associative memories. Different attractors of the network will be identified…

In this paper, we introduce variational semantic memory into meta-learning to acquire long-term knowledge for few-shot learning. The variational semantic memory accrues and stores semantic information for the probabilistic inference of…

We consider the problem of neural association for a network of non-binary neurons. Here, the task is to first memorize a set of patterns using a network of neurons whose states assume values from a finite number of integer levels. Later,…

A famous biologically inspired hierarchical model firstly proposed by Riesenhuber and Poggio has been successfully applied to multiple visual recognition tasks. The model is able to achieve a set of position- and scale-tolerant recognition,…

The brain is targeted for processing temporal sequence information. It remains largely unclear how the brain learns to store and retrieve sequence memories. Here, we study how recurrent networks of binary neurons learn sequence attractors…

Recent work in continual learning has highlighted the beneficial effect of resampling weights in the last layer of a neural network (``zapping"). Although empirical results demonstrate the effectiveness of this approach, the underlying…