Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks. However, their extensive memory requirements, particularly due to KV cache growth during long-text understanding and generation, present significant challenges for deployment in resource-constrained environments. Quantization has emerged as a promising solution to reduce memory consumption while preserving historical information. We propose XQuant, a training-free and plug-and-play framework that achieves ultra-low equivalent bit-width KV cache quantization. XQuant introduces two key innovations: a computationally negligible data-free calibration method and cross-layer KV cache compression, enabling quantization to sub-1.4 bits. Extensive experiments on TruthfulQA and LongBench demonstrate that XQuant outperforms state-of-the-art methods (e.g., KIVI-2bit and AsymKV-1.5bit) by achieving lower bit-width while maintaining superior performance, establishing a better trade-off between memory efficiency and model accuracy.
@article{arxiv.2510.11236,
title = {XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression},
author = {Haoqi Yang and Yao Yao and Zuchao Li and Baoyuan Qi and Guoming Liu and Hai Zhao},
journal= {arXiv preprint arXiv:2510.11236},
year = {2025}
}
Comments
To be published in The 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)