English

Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data

Machine Learning 2024-07-03 v2 Artificial Intelligence Image and Video Processing

Abstract

Lossy compression has become an important technique to reduce data size in many domains. This type of compression is especially valuable for large-scale scientific data, whose size ranges up to several petabytes. Although Autoencoder-based models have been successfully leveraged to compress images and videos, such neural networks have not widely gained attention in the scientific data domain. Our work presents a neural network that not only significantly compresses large-scale scientific data, but also maintains high reconstruction quality. The proposed model is tested with scientific benchmark data available publicly and applied to a large-scale high-resolution climate modeling data set. Our model achieves a compression ratio of 140 on several benchmark data sets without compromising the reconstruction quality. 2D simulation data from the High-Resolution Community Earth System Model (CESM) Version 1.3 over 500 years are also being compressed with a compression ratio of 200 while the reconstruction error is negligible for scientific analysis.

Keywords

Cite

@article{arxiv.2307.04216,
  title  = {Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data},
  author = {Hieu Le and Jian Tao},
  journal= {arXiv preprint arXiv:2307.04216},
  year   = {2024}
}

Comments

14 pages

R2 v1 2026-06-28T11:25:28.333Z