Error-bounded lossy compression has been effective in significantly reducing the data storage/transfer burden while preserving the reconstructed data fidelity very well. Many error-bounded lossy compressors have been developed for a wide range of parallel and distributed use cases for years. They are designed with distinct compression models and principles, such that each of them features particular pros and cons. In this paper we provide a comprehensive survey of emerging error-bounded lossy compression techniques. The key contribution is fourfold. (1) We summarize a novel taxonomy of lossy compression into 6 classic models. (2) We provide a comprehensive survey of 10 commonly used compression components/modules. (3) We summarized pros and cons of 46 state-of-the-art lossy compressors and present how state-of-the-art compressors are designed based on different compression techniques. (4) We discuss how customized compressors are designed for specific scientific applications and use-cases. We believe this survey is useful to multiple communities including scientific applications, high-performance computing, lossy compression, and big data.
@article{arxiv.2404.02840,
title = {A Survey on Error-Bounded Lossy Compression for Scientific Datasets},
author = {Sheng Di and Jinyang Liu and Kai Zhao and Xin Liang and Robert Underwood and Zhaorui Zhang and Milan Shah and Yafan Huang and Jiajun Huang and Xiaodong Yu and Congrong Ren and Hanqi Guo and Grant Wilkins and Dingwen Tao and Jiannan Tian and Sian Jin and Zizhe Jian and Daoce Wang and MD Hasanur Rahman and Boyuan Zhang and Shihui Song and Jon C. Calhoun and Guanpeng Li and Kazutomo Yoshii and Khalid Ayed Alharthi and Franck Cappello},
journal= {arXiv preprint arXiv:2404.02840},
year = {2025}
}
Comments
This paper has been submitted to ACM Computing journal. This is a second-stage revised version based on review comments