Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragmatic solution for preventing OOV, especially in the context of larger models. However, breaking a character down to individual bytes significantly increases the sequence length for long-tail tokens in languages such as Chinese, Japanese, and Korean (CJK) and other character-diverse contexts such as emoji. The increased sequence length results in longer computation during both training and inference. In this work, we propose a simple compression technique that reduces the sequence length losslessly.
Cite
@article{arxiv.2506.07541,
title = {Bit-level BPE: Below the byte boundary},
author = {Sangwhan Moon and Tatsuya Hiraoka and Naoaki Okazaki},
journal= {arXiv preprint arXiv:2506.07541},
year = {2025}
}