Unlocking Noisy Real-World Corpora for Foundation Model Pre-Training via Quality-Aware Tokenization
Abstract
Current tokenization methods process sequential data without accounting for signal quality, limiting their effectiveness on noisy real-world corpora. We present QA-Token (Quality-Aware Tokenization), which incorporates data reliability directly into vocabulary construction. We make three key contributions: (i) a bilevel optimization formulation that jointly optimizes vocabulary construction and downstream performance, (ii) a reinforcement learning approach that learns merge policies through quality-aware rewards with convergence guarantees, and (iii) an adaptive parameter learning mechanism via Gumbel-Softmax relaxation for end-to-end optimization. Our experimental evaluation demonstrates consistent improvements: genomics (6.7 percentage point F1 gain in variant calling over BPE), finance (30% Sharpe ratio improvement). At foundation scale, we tokenize a pretraining corpus comprising 1.7 trillion base-pairs and achieve state-of-the-art pathogen detection (94.53 MCC) while reducing token count by 15%. We unlock noisy real-world corpora, spanning petabases of genomic sequences and terabytes of financial time series, for foundation model training with zero inference overhead.
Keywords
Cite
@article{arxiv.2602.06394,
title = {Unlocking Noisy Real-World Corpora for Foundation Model Pre-Training via Quality-Aware Tokenization},
author = {Arvid E. Gollwitzer and Paridhi Latawa and David de Gruijl and Deepak A. Subramanian and Adrián Noriega de la Colina},
journal= {arXiv preprint arXiv:2602.06394},
year = {2026}
}