Related papers: Automated data reduction workflows for astronomy
Processing of raw data from modern astronomical instruments is nowadays often carried out using dedicated software, so-called "pipelines" which are largely run in automated operation. In this paper we describe the data reduction pipeline of…
Scientific communities naturally tend to organize around data ecosystems created by the combination of their observational devices, their data repositories, and the workflows essential to carry their research from observation to discovery.…
Kepler's primary mission is a search for earth-size exoplanets in the habitable zone of late-type stars using the transit method. To effectively accomplish this mission, Kepler orbits the Sun and stares nearly continuously at one…
Wavelets have been used extensively for several years now in astronomy for many purposes, ranging from data filtering and deconvolution, to star and galaxy detection or cosmic ray removal. More recent sparse representations such ridgelets…
KMOS is a multi-object near-infrared integral field spectrometer with 24 deployable cryogenic pick-off arms. Inevitably, data processing is a complex task that requires careful calibration and quality control. In this paper we describe all…
Research process automation -- the reliable, efficient, and reproducible execution of linked sets of actions on scientific instruments, computers, data stores, and other resources -- has emerged as an essential element of modern science. We…
Data intensive applications often involve the analysis of large datasets that require large amounts of compute and storage resources. While dedicated compute and/or storage farms offer good task/data throughput, they suffer low resource…
ADARI (Astronomical DAta Reporting Infrastructure) is a system designed for creating graphical reports of astronomical data so that the quality of these products can be assessed. It has been designed from the ground up to be…
We present Easylife, the software environment developed within the framework of the VIPERS project for automatic data reduction and survey handling. Easylife is a comprehensive system to automatically reduce spectroscopic data, to monitor…
Workflow technology is rapidly evolving and, rather than being limited to modeling the control flow in business processes, is becoming a key mechanism to perform advanced data management, such as big data analytics. This survey focuses on…
Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all…
Scientific research in many fields routinely requires the analysis of large datasets, and scientists often employ workflow systems to leverage clusters of computers for their data analysis. However, due to their size and scale, these…
Stimela2 is a new-generation framework for developing data reduction workflows. It is designed for radio astronomy data but can be adapted for other data processing applications. Stimela2 aims at the middle ground between ease of…
Fast variability of optical objects is an interesting though poorly explored subject in modern astronomy. Real-time data processing and identification of transient celestial events in the images is very important for such study as it allows…
The amount of collected data in many scientific fields is increasing, all of them requiring a common task: extract knowledge from massive, multi parametric data sets, as rapidly and efficiently possible. This is especially true in astronomy…
Earth observation analytics have the potential to transform many sectors. However, due to limited ground connections, it currently takes hours to days to download and analyze Earth observation data, diminishing the value of data for…
Workflow is a common term used to describe a systematic breakdown of tasks that need to be performed to solve a problem. This concept has found best use in scientific and business applications for streamlining and improving the performance…
The era of data-intensive astronomy is being ushered in with the increasing size and complexity of observational data across wavelength and time domains, the development of algorithms to extract information from this complexity, and the…
Progress in science is deeply bound to the effective use of high-performance computing infrastructures and to the efficient extraction of knowledge from vast amounts of data. Such data comes from different sources that follow a cycle…
DBSP_DRP is a python package that provides fully automated data reduction of data taken by the Double Spectrograph (DBSP) at the 200-inch Hale Telescope at Palomar Observatory (Oke & Gunn, 1982). The underlying data reduction functionality…