Related papers: Optimising the Processing and Storage of Visibilit…
The growing volume of scientific simulation data presents a significant challenge for storage and transfer. Error-bounded lossy compression has emerged as a critical solution for mitigating these challenges, providing a means to reduce data…
This paper presents a new algorithm for the lossy compression of scalar data defined on 2D or 3D regular grids, with topological control. Certain techniques allow users to control the pointwise error induced by the compression. However, in…
Lossy compression algorithms take advantage of the inherent limitations of the human eye and discard information that cannot be seen. In the present paper, a technique termed as Lossy Data Compression using Logarithm (LDCL) is proposed to…
Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all…
In order to generate credible 0.1-2 {\mu}m SEDs, the GAMA project requires many Gigabytes of imaging data from a number of instruments to be re-processed into a standard format. In this paper we discuss the software infrastructure we use,…
Gamma-ray astronomy is able to acquire large data volumes that astronomers use to draw scientific conclusions from. Ensuring the possibility of accessing and utilizing this data also after the lifetime of currently running experiments…
As gravitational wave detectors improve in sensitivity, signal-to-noise ratios of compact binary coalescences will dramatically increase, reaching values in the hundreds and potentially thousands. Such strong signals offer both exciting…
The data compression technology now is fully developed and widely used in many fields such as communication, multi-media, image information processing and so on. The large physical experiments, especially the ones with Micro-pattern Gas…
Robust correlation estimation is essential in high-dimensional settings, particularly when data are contaminated by outliers or exhibit heavy-tailed behavior. Many robust loss functions of practical interest-such as those involving…
Intrinsic alignments of galaxies can mimic to an extent the effects of shear caused by weak gravitational lensing. Previous studies have shown that for shallow surveys with median redshifts z_m = 0.1, the intrinsic alignment dominates the…
We investigate data compression schemes for proposed all-sky diffraction-limited visible/NIR sky surveys aimed at the dark energy problem. We show that lossy square-root compression to 1 bit of noise per pixel, followed by standard lossless…
Over the past decades, there has been an explosion in the amount of available Earth Observation (EO) data. The unprecedented coverage of the Earth's surface and atmosphere by satellite imagery has resulted in large volumes of data that must…
The truncated singular value decomposition (SVD) of the measurement matrix is the optimal solution to the_representation_ problem of how to best approximate a noisy measurement matrix using a low-rank matrix. Here, we consider the…
Next generation radio interferometric telescopes are entering an era of big data with extremely large data sets. While these telescopes can observe the sky in higher sensitivity and resolution than before, computational challenges in image…
Compressed sensing deals with the reconstruction of sparse signals using a small number of linear measurements. One of the main challenges in compressed sensing is to find the support of a sparse signal. In the literature, several bounds on…
The peculiar motion of galaxies can be a particularly sensitive probe of gravitational collapse. As such, it can be used to measure the dynamics of dark matter and dark energy as well the nature of the gravitational laws at play on…
We present a method for radical linear compression of datasets where the data are dependent on some number $M$ of parameters. We show that, if the noise in the data is independent of the parameters, we can form $M$ linear combinations of…
This paper proposes a new end-to-end trainable model for lossy image compression, which includes several novel components. The method incorporates 1) an adequate perceptual similarity metric; 2) saliency in the images; 3) a hierarchical…
The domain of radio astronomy is currently facing significant computational challenges, foremost amongst which are those posed by the development of the world's largest radio telescope, the Square Kilometre Array (SKA). Preliminary…
Today's scientific high performance computing (HPC) applications or advanced instruments are producing vast volumes of data across a wide range of domains, which introduces a serious burden on data transfer and storage. Error-bounded lossy…