Related papers: A data model to connect the ESO Data Processing Sy…

Astronomical data reduction is usually done with processing pipelines that consist of a series of individual processing steps that can be executed stand-alone. These processing steps are then strung together into workflows and fed with data…

The recently released Reflex scientific workflow environment supports the interactive execution of ESO VLT data reduction pipelines. Reflex is based upon the Kepler workflow engine, and provides components for organising the data, executing…

METIS, the Mid-Infrared ELT Imager and Spectrograph, is one of the four first-generation ELT instruments scheduled to see first light in 2028. Its two main science modules are supported by an adaptive optics system featuring a pyramid…

Written in Python and utilising ParselTongue to interface with the Astronomical Image Processing System (AIPS), the e-MERLIN data reduction pipeline is intended to automate the procedures required in processing and calibrating radio…

Modern in-orbit satellites and other available remote sensing tools have generated a huge availability of public data waiting to be exploited in different formats hosted on different servers. In this context, ETL formalism becomes relevant…

Processing of raw data from modern astronomical instruments is nowadays often carried out using dedicated software, so-called "pipelines" which are largely run in automated operation. In this paper we describe the data reduction pipeline of…

METIS, the Mid-infrared ELT Imager and Spectrograph (formerly called MIDIR), is a proposed instrument for the European Extremely Large Telescope (E-ELT), currently undergoing a phase-A study. The study is carried out within the framework of…

Currently, a variety of pipeline tools are available for use in data engineering. Data scientists can use these tools to resolve data wrangling issues associated with data and accomplish some data engineering tasks from data ingestion…

This paper provides a case study on satellite data processing, storage, and distribution in the space weather domain by introducing the Satellite Data Downloading System (SDDS). The approach proposed in this paper was evaluated through…

Modern ETL streaming pipelines extract data from various sources and forward it to multiple consumers, such as data warehouses (DW) and analytical systems that leverage machine learning (ML). However, the increasing number of systems that…

Developer Productivity Dashboards are essential for visualizing DevOps performance metrics such as Deployment Frequency and Change Failure Rate (DORA). However, the utility of these dashboards is frequently undermined by data reliability…

We show that the standard computational pipeline of probabilistic programming systems (PPSs) can be inefficient for estimating expectations and introduce the concept of expectation programming to address this. In expectation programming,…

The Mid-infrared ELT Imager and Spectrograph (METIS) is a first-generation instrument for the Extremely Large Telescope (ELT), Europe's next-generation 39 m ground-based telescope for optical and infrared wavelengths. METIS will offer…

The amdlib AMBER data reduction software is meant to produce AMBER data products from the raw data files that are sent to the PIs of different proposals or that can be found in the ESO data archive. The way defined by ESO to calibrate the…

The ESPRESSO instrument, to be commissioned in the next months at the ESO VLT, is bound to became a landmark in the field of high-resolution optical spectroscopy, both for its ground-breaking science objectives (search for Earth-like…

Currently public ESO data sets pertinent to the CDFS/GOODS field are briefly illustrated along with an indication on how to get access to them. Future ESO plans for complementing the GOODS database with optical/IR imaging and optical…

Users of astronomical observatories rely on Exposure Time Calculators (ETC) to prepare their proposals and then their observations. The ETC is therefore a crucial element in an observatory's data workflow and in particular is key to…

Due to the Internet of Everything (IoE), data generated in our life become larger. As a result, we need more effort to analyze the data and extract valuable information. In the cloud computing environment, all data analysis is done in the…

The ATLAS Event Streaming Service (ESS) at the LHC is an approach to preprocess and deliver data for Event Service (ES) that has implemented a fine-grained approach for ATLAS event processing. The ESS allows one to asynchronously deliver…

The Extract, Transform, Load (ETL) workflow is fundamental for populating and maintaining data warehouses and other data stores accessed by analysts for downstream tasks. A major shortcoming of modern ETL solutions is the extensive need for…