Related papers: Building a scalable global data processing pipelin…
We describe a compression method for floating-point astronomical images that gives compression ratios of 6 -- 10 while still preserving the scientifically important information in the image. The pixel values are first preprocessed by…
New approaches for data provenance and data management (DPDM) are required for mega science projects like the Square Kilometer Array, characterized by extremely large data volume and intense data rates, therefore demanding innovative and…
The modernization of the Common Agricultural Policy (CAP) requires the large scale and frequent monitoring of agricultural land. Towards this direction, the free and open satellite data (i.e., Sentinel missions) have been extensively used…
Proper quality control (QC) is time consuming when working with large-scale medical imaging datasets, yet necessary, as poor-quality data can lead to erroneous conclusions or poorly trained machine learning models. Most efforts to reduce…
Astronomy has always been at the forefront of information technology, moving from the era of photographic plates, to digital snapshots and now to digital movies of the sky. This has brought about a data explosion with multi- terabyte…
We introduce NebulOS, a Big Data platform that allows a cluster of Linux machines to be treated as a single computer. With NebulOS, the process of writing a massively parallel program for a datacenter is no more complicated than writing a…
The amount of remote sensing data available to applications is constantly growing due to the rise of very-high-resolution sensors and short repeat cycle satellites. Consequently, tackling computational complexity in Earth Observation…
The VISIONS public survey provides large-scale, multiepoch imaging of five nearby star-forming regions at subarcsecond resolution in the near-infrared. All data collected within the program and provided by the European Southern Observatory…
The FITS is the standard file format in astronomy, and it has been extended to agree with astronomical needs of the day. However, astronomical datasets have been inflating year by year. In case of ALMA telescope, a ~ TB scale 4-dimensional…
Astronomy depends on ever increasing computing power. Processor clock-rates have plateaued, and increased performance is now appearing in the form of additional processor cores on a single chip. This poses significant challenges to the…
The production of science-ready data from major solar telescopes requires expertise beyond that of the typical observer. This is a consequence of the increasing complexity of instruments and observing sequences, which require calibrations…
KMOS is a multi-object near-infrared integral field spectrometer with 24 deployable cryogenic pick-off arms. Inevitably, data processing is a complex task that requires careful calibration and quality control. In this paper we describe all…
As members of the instrument team for the Advanced CCD Imaging Spectrometer (ACIS) on NASA's Chandra X-ray Observatory and as Chandra General Observers, we have developed a wide variety of data analysis methods that we believe are useful to…
Big earth science data offers the scientific community great opportunities. Many more studies at large-scales, over long-terms and at high resolution can now be conducted using the rich information collected by remote sensing satellites,…
This chapter describes how astronomical imaging survey data have become a vital part of modern astronomy, how these data are archived and then served to the astronomical community through on-line data access portals. The Virtual…
In the current data-driven science era, it is needed that data analysis techniques has to quickly evolve to face with data whose dimensions has increased up to the Petabyte scale. In particular, being modern astrophysics based on…
Cyber-Physical Systems (CPSs) employed for Industrial Automation often require the adoption of a hybrid data processing approach mediating between cloud, edge, and fog computing paradigms. Nowadays, it is possible to shift data…
The primary use of technical computing in the oil and gas industries is for seismic imaging of the earth's subsurface, driven by the business need for making well-informed drilling decisions during petroleum exploration and production.…
At the Canadian Astronomy Data Centre, we have combined our cloud computing system, CANFAR, with the world's most advanced machine learning software, Skytree, to create the world's first cloud computing system for data mining in astronomy.…
Cyber-Physical Systems (CPSs) have been pervasive including smart grid, autonomous automobile systems, medical monitoring, process control systems, robotics systems, and automatic pilot avionics. As usually implemented on embedded devices,…