Related papers: The Vera C. Rubin Observatory Data Butler and Pipe…
The Vera C. Rubin Observatory, through its Legacy Survey of Space and Time, will soon start producing 10 million alerts on transient astronomical objects per night. Due to logistics and bandwidth, alerts will not be dispatched directly to…
Collimator detection remains a challenging task in X-ray systems with unreliable or non-available information about the detectors position relative to the source. This paper presents a physically motivated image processing pipeline for…
Fast variability of optical objects is an interesting though poorly explored subject in modern astronomy. Real-time data processing and identification of transient celestial events in the images is very important for such study as it allows…
The most common approach to implementing data analysis pipelines involves obtaining point estimates from the upstream modules and then treating these as known quantities when working with the downstream ones. This approach is…
Observational astronomy has changed drastically in the last decade: manually driven target-by-target instruments have been replaced by fully automated robotic telescopes. Data acquisition methods have advanced to the point that terabytes of…
The R-package REPPlab is designed to explore multivariate data sets using one-dimensional unsupervised projection pursuit. It is useful in practice as a preprocessing step to find clusters or as an outlier detection tool for multivariate…
We implemented a real-time data processor (rta-dp) framework that can be used to develop real-time analysis pipelines and data handling systems to manage high-throughput data streams with distributed applications in the context of ground…
We present MapReader, a free, open-source software library written in Python for analyzing large map collections (scanned or born-digital). This library transforms the way historians can use maps by turning extensive, homogeneous map sets…
The robot manipulation ecosystem currently faces issues with integrating open-source components and reproducing results. This limits the ability of the community to benchmark and compare the performance of different solutions to one another…
Researchers have identified datasets used for training computer vision (CV) models as an important source of hazardous outcomes, and continue to examine popular CV datasets to expose their harms. These works tend to treat datasets as…
We describe principal components of the new spectroscopic data pipeline for the multi-object MMT/Magellan Infrared Spectrograph (MMIRS). The pipeline is implemented in IDL and C++. The performance of the data processing algorithms is…
Virtual representations of physical critical infrastructures, such as water or energy plants, are used for simulations and digital twins to ensure resilience and continuity of their services. These models usually require 3D point clouds…
Executing machine learning (ML) pipelines in real-time on radiology images is hard due to the limited computing resources in clinical environments and the lack of efficient data transfer capabilities to run them on research clusters. We…
Data pre-processing pipelines are the bread and butter of any successful AI project. We introduce a novel programming model for pipelines in a data lakehouse, allowing users to interact declaratively with assets in object storage. Motivated…
The HIFI data processing pipeline was developed to systematically process diagnostic, calibration and astronomical observations taken with the HIFI science instrumentas part of the Herschel mission. The HIFI pipeline processed data from all…
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, iden- tify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and…
Baikal-GVD is a gigaton-scale neutrino observatory under construction in Lake Baikal. It currently produces about 100 GB of data every day. For their automatic processing, the Baikal Analysis and Reconstruction software (BARS) was…
Parallel dataflow systems are a central part of most analytic pipelines for big data. The iterative nature of many analysis and machine learning algorithms, however, is still a challenge for current systems. While certain types of bulk…
Data processing pipelines represent an important slice of the astronomical software library that include chains of processes that transform raw data into valuable information via data reduction and analysis. In this work we present Corral,…
In the era of multi-messenger astronomy the exploration of the early emission from transients is key for understanding the encoded physics. At the same time, current generation networks of fully-robotic telescopes provide new opportunities…