Large token-indexed lookup tables provide a compute-decoupled scaling path, but their practical gains are often limited by poor parameter efficiency and rapid memory growth. We attribute these limitations to Zipfian under-training of the long tail, heterogeneous demand across layers, and "slot collapse" that produces redundant embeddings. To address this, we propose X-GRAM, a frequency-aware dynamic token-injection framework. X-GRAM employs hybrid hashing and alias mixing to compress the tail while preserving head capacity, and refines retrieved vectors via normalized SwiGLU ShortConv to extract diverse local n-gram features. These signals are integrated into attention value streams and inter-layer residuals using depth-aware gating, effectively aligning static memory with dynamic context. This design introduces a memory-centric scaling axis that decouples model capacity from FLOPs. Extensive evaluations at the 0.73B and 1.15B scales show that X-GRAM improves average accuracy by as much as 4.4 points over the vanilla backbone and 3.2 points over strong retrieval baselines, while using substantially smaller tables in the 50% configuration. Overall, by decoupling capacity from compute through efficient memory management, X-GRAM offers a scalable and practical paradigm for future memory-augmented architectures. Code aviliable in https://github.com/Longyichen/X-gram.
@article{arxiv.2604.21724,
title = {Beyond N-gram: Data-Aware X-GRAM Extraction for Efficient Embedding Parameter Scaling},
author = {Yilong Chen and Yanxi Xie and Zitian Gao and He Xin and Yihao Xiao and Jason Klein Liu and Haoming Luo and Yifan Luo and Zhengmao Ye and Tingwen Liu and Xin Zhao and Ran Tao and Bryan Dai},
journal= {arXiv preprint arXiv:2604.21724},
year = {2026}
}