Memory Mosaics
Machine Learning
2025-03-03 v3 Artificial Intelligence
Neural and Evolutionary Computing
Abstract
Memory Mosaics are networks of associative memories working in concert to achieve a prediction task of interest. Like transformers, memory mosaics possess compositional capabilities and in-context learning capabilities. Unlike transformers, memory mosaics achieve these capabilities in comparatively transparent way ("predictive disentanglement"). We illustrate these capabilities on a toy example and also show that memory mosaics perform as well or better than transformers on medium-scale language modeling tasks.
Keywords
Cite
@article{arxiv.2405.06394,
title = {Memory Mosaics},
author = {Jianyu Zhang and Niklas Nolte and Ranajoy Sadhukhan and Beidi Chen and Léon Bottou},
journal= {arXiv preprint arXiv:2405.06394},
year = {2025}
}