Related papers: A compression scheme for radio data in high perfor…
Increasing data volumes delivered by a new generation of radio interferometers require computationally efficient and robust calibration algorithms. In this paper, we propose distributed calibration as a way of improving both computational…
An important application of next-generation wide-field radio interferometers is making high dynamic range maps of radio emission. Traditional deconvolution methods like CLEAN can give poor recovery of diffuse structure, prompting the…
Future large scale cosmological surveys will provide huge data sets whose analysis requires efficient data compression. Calculating accurate covariances is extremely challenging with increasing number of statistics used. Here we introduce a…
Error-controlled lossy compressors have been widely used in scientific applications to reduce the unprecedented size of scientific data while keeping data distortion within a user-specified threshold. While they significantly mitigate the…
Lossy compressors are increasingly adopted in scientific research, tackling volumes of data from experiments or parallel numerical simulations and facilitating data storage and movement. In contrast with the notion of entropy in lossless…
We describe the data processing pipeline of the Planck Low Frequency Instrument (LFI) data processing centre (DPC) to create and characterize full-sky maps based on the first 15.5 months of operations at 30, 44 and 70 GHz. In particular, we…
New and upcoming radio interferometers will produce unprecedented amounts of data that demand extremely powerful computers for processing. This is a limiting factor due to the large computational power and energy costs involved. Such…
High-energy, large-scale particle colliders in nuclear and high-energy physics generate data at extraordinary rates, reaching up to $1$ terabyte and several petabytes per second, respectively. The development of real-time, high-throughput…
The latest generation of radio astronomy interferometers will conduct all sky surveys with data products consisting of petabytes of spectral line data. Traditional approaches to identifying and parameterising the astrophysical sources…
Bringing a high-dimensional dataset into science-ready shape is a formidable challenge that often necessitates data compression. Compression has accordingly become a key consideration for contemporary cosmology, affecting public data…
Next generation radio telescopes, like the Square Kilometre Array, will acquire an unprecedented amount of data for radio astronomy. The development of fast, parallelisable or distributed algorithms for handling such large-scale data sets…
The next-generation radio astronomy instruments are providing a massive increase in sensitivity and coverage, through increased stations in the array and frequency span. Two primary problems encountered when processing the resultant…
Compression algorithms reduce the redundancy in data representation to decrease the storage required for that data. Data compression offers an attractive approach to reducing communication costs by using available bandwidth effectively.…
We are performing a transient, microsecond timescale radio sky survey, called "Astropulse," using the Arecibo telescope. Astropulse searches for brief (0.4 {\mu}s to 204.8 {\mu}s), wideband (relative to its 2.5 MHz bandwidth) radio pulses…
A new phase-coherent technique for the calibration of polarimetric data is presented. Similar to the one-dimensional form of convolution, data are multiplied by the response function in the frequency domain. Therefore, the system response…
We describe and compare several post-correlation radio frequency interference classification methods. As data sizes of observations grow with new and improved telescopes, the need for completely automated, robust methods for radio frequency…
Real-world data typically contain repeated and periodic patterns. This suggests that they can be effectively represented and compressed using only a few coefficients of an appropriate basis (e.g., Fourier, Wavelets, etc.). However, distance…
It is known that data rates in standard cellular networks are limited due to inter-cell interference. An effective solution of this problem is to use the multi-cell cooperation idea. In Cloud Radio Access Network, which is a candidate…
Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all…
This work studies distributed compression for the uplink of a cloud radio access network where multiple multi-antenna base stations (BSs) are connected to a central unit, also referred to as cloud decoder, via capacity-constrained backhaul…