LLMZip: Lossless Text Compression using Large Language Models
Information Theory
2023-06-28 v2 Computation and Language
Machine Learning
math.IT
Abstract
We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens. This estimate is significantly smaller than currently available estimates in \cite{cover1978convergent}, \cite{lutati2023focus}. A natural byproduct is an algorithm for lossless compression of English text which combines the prediction from the large language model with a lossless compression scheme. Preliminary results from limited experiments suggest that our scheme outperforms state-of-the-art text compression schemes such as BSC, ZPAQ, and paq8h.
Cite
@article{arxiv.2306.04050,
title = {LLMZip: Lossless Text Compression using Large Language Models},
author = {Chandra Shekhara Kaushik Valmeekam and Krishna Narayanan and Dileep Kalathil and Jean-Francois Chamberland and Srinivas Shakkottai},
journal= {arXiv preprint arXiv:2306.04050},
year = {2023}
}
Comments
7 pages, 4 figures, 4 tables, preprint, added results on using LLMs with arithmetic coding