We present MELODI, a novel memory architecture designed to efficiently process long documents using short context windows. The key principle behind MELODI is to represent short-term and long-term memory as a hierarchical compression scheme across both network layers and context windows. Specifically, the short-term memory is achieved through recurrent compression of context windows across multiple layers, ensuring smooth transitions between windows. In contrast, the long-term memory performs further compression within a single middle layer and aggregates information across context windows, effectively consolidating crucial information from the entire history. Compared to a strong baseline - the Memorizing Transformer employing dense attention over a large long-term memory (64K key-value pairs) - our method demonstrates superior performance on various long-context datasets while remarkably reducing the memory footprint by a factor of 8.
@article{arxiv.2410.03156,
title = {MELODI: Exploring Memory Compression for Long Contexts},
author = {Yinpeng Chen and DeLesley Hutchins and Aren Jansen and Andrey Zhmoginov and David Racz and Jesper Andersen},
journal= {arXiv preprint arXiv:2410.03156},
year = {2024}
}