Tokenisation is NP-Complete
Data Structures and Algorithms
2024-12-20 v1 Computation and Language
Formal Languages and Automata Theory
Abstract
In this work, we prove the NP-completeness of two variants of tokenisation, defined as the problem of compressing a dataset to at most symbols by either finding a vocabulary directly (direct tokenisation), or selecting a sequence of merge operations (bottom-up tokenisation).
Cite
@article{arxiv.2412.15210,
title = {Tokenisation is NP-Complete},
author = {Philip Whittington and Gregor Bachmann and Tiago Pimentel},
journal= {arXiv preprint arXiv:2412.15210},
year = {2024}
}