Related papers: Polynomial data compression for large-scale physic…
A new approach to data compression is developed and applied to multimedia content. This method separates messages into components suitable for both lossless coding and 'lossy' or statistical coding techniques, compressing complex objects by…
Very High Energy gamma-ray astronomy with the Cherenkov Telescope Array (CTA) is evolving towards the model of a public observatory. Handling, processing and archiving the large amount of data generated by the CTA instruments and delivering…
The last 20 years have seen the development of new techniques in Astroparticle Physics providing access to the highest end of the electromagnetic spectrum. It has been shown that some sources emit photons up to energies close to 100 TeV.…
We review some aspects of the current state of data-intensive astronomy, its methods, and some outstanding data analysis challenges. Astronomy is at the forefront of "big data" science, with exponentially growing data volumes and data…
Data compression has been widely applied in many data processing areas. Compression methods use variable-size codes with the shorter codes assigned to symbols or groups of symbols that appear in the data frequently. Fibonacci coding, as a…
We introduce a new protocol for a lossy data compression algorithm which is based on constraint satisfaction gates. We show that the theoretical capacity of algorithms built from standard parity-check gates converges exponentially fast to…
Recent advances in signal processing have focused on the use of sparse representations in various applications. A new field of interest based on sparsity has recently emerged: compressed sensing. This theory is a new sampling framework that…
This paper proposes a novel entropy encoding technique for lossless data compression. Representing a message string by its lexicographic index in the permutations of its symbols results in a compressed version matching Shannon entropy of…
Modern data compression methods are slowly reaching their limits after 80 years of research, millions of papers, and wide range of applications. Yet, the extravagant 6G communication speed requirement raises a major open question for…
The Cherenkov Telescopic Array (CTA), the next-generation ground-based gamma-ray observatory, will have unprecedented sensitivity, providing answers to open questions in gamma-ray cosmology and fundamental physics. Using simulations of…
The era of data-intensive astronomy is being ushered in with the increasing size and complexity of observational data across wavelength and time domains, the development of algorithms to extract information from this complexity, and the…
Modern scientific simulations, observations, and large-scale experiments generate data at volumes that often exceed the limits of storage, processing, and analysis. This challenge drives the development of data reduction methods that…
We present a method for radical linear compression of datasets where the data are dependent on some number $M$ of parameters. We show that, if the noise in the data is independent of the parameters, we can form $M$ linear combinations of…
Future large scale cosmological surveys will provide huge data sets whose analysis requires efficient data compression. Calculating accurate covariances is extremely challenging with increasing number of statistics used. Here we introduce a…
In recent years, ground-based very-high-energy (VHE; E>100 GeV) gamma-ray astronomy has experienced a major breakthrough with the impressive astrophysical results obtained mainly by the current generation experiments like H.E.S.S., MAGIC,…
Frugal computing is becoming an important topic for environmental reasons. In this context, several techniques have been proposed to reduce the storage of scientific data by dedicated compression methods specially tailored for arrays of…
Compression algorithms reduce the redundancy in data representation to decrease the storage required for that data. Data compression offers an attractive approach to reducing communication costs by using available bandwidth effectively.…
Radio-astronomy is about to embark on a new way of doing science. The revolution that is about to take place is not due to the enormous sensitivity of the Square Kilometre Array, which is still a decade away, but due to its pathfinders,…
Physics experiments produce enormous amount of raw data, counted in petabytes per day. Hence, there is large effort to reduce this amount, mainly by using some filters. The situation can be improved by additionally applying some data…
Large scale numerical experiments are commonplace today in theoretical physics. The high performance algorithms described herein are the most compact, efficient methods known for representing and analyzing systems modeled well by sets or…