Related papers: Pipeline Processing of VLBI Data
This paper presents an innovative machine learning methodology that leverages on long-term vibroacoustic measurements to perform automated predictions of the needed pigging operations in crude oil trunklines. Historical pressure signals…
We develop a tool called PipeGen for efficient data transfer between database management systems (DBMSs). PipeGen targets data analytics workloads on shared-nothing engines. It supports scenarios where users seek to perform different parts…
The Gemini Planet Imager Exoplanet Survey (GPIES) is a multi-year direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process…
Data processing pipelines need to be executed at scales ranging from small runs up through large production data release runs resulting in millions of data products. As part of the Rubin Observatory's pipeline execution system, BPS is the…
The US National Park Service (NPS) assesses the night sky quality over parks by capturing a series of overlapping images to obtain a mosaic view of the entire night sky. The NPS Night Skies Program has integrated a sequence of scripts and…
Data scientists develop ML pipelines in an iterative manner: they repeatedly screen a pipeline for potential issues, debug it, and then revise and improve its code according to their findings. However, this manual process is tedious and…
Semi-implicit variational inference (SIVI) enriches the expressiveness of variational families by utilizing a kernel and a mixing distribution to hierarchically define the variational distribution. Existing SIVI methods parameterize the…
Multi-product pipelines are a highly efficient means of transporting liquids. Traditionally used to transport petroleum, its products and derivatives, they are now being repurposed to transport liquified natural gas admixed with hydrogen of…
With the arrival of a number of wide-field snapshot image-plane radio transient surveys, there will be a huge influx of images in the coming years making it impossible to manually analyse the datasets. Automated pipelines to process the…
Data pipelines are essential in stream processing as they enable the efficient collection, processing, and delivery of real-time data, supporting rapid data analysis. In this paper, we present AutoStreamPipe, a novel framework that employs…
The increasing adoption of digital health technologies has amplified the need for robust, interoperable solutions to manage complex healthcare data. We present the Spezi Data Pipeline, an open-source Python toolkit designed to streamline…
Particle Image Velocimetry (PIV) estimates velocities through correlations of particle images within interrogation windows, leading to a spatial modulation of the velocity field. Although in principle Particle Tracking Velocimetry (PTV)…
Machine learning (ML) offers powerful methods for detecting and modeling associations often in data with large feature spaces and complex associations. Many useful tools/packages (e.g. scikit-learn) have been developed to make the various…
We have constructed a new, fast, robust and reliable pipeline to detect variable stars from the ROTSE-IIId archival data. Turkish share of ROTSE-III archive contains approximately one million objects from a large field of view (1.85\dgr)…
SPREV, short for hyperSphere Reduced to two-dimensional Regular Polygon for Visualisation, is a novel dimensionality reduction technique developed to address the challenges of reducing dimensions and visualizing labeled datasets that…
Recent work has made significant progress in helping users to automate single data preparation steps, such as string-transformations and table-manipulation operators (e.g., Join, GroupBy, Pivot, etc.). We in this work propose to automate…
LiDAR are increasingly being used in intelligent vehicles (IV) or intelligent transportation systems (ITS). Storage and transmission of data generated by LiDAR sensors are one of the most challenging aspects of their deployment. In this…
We present a final description of the data-processing pipeline for the Planck, Low Frequency Instrument (LFI), implemented for the 2018 data release. Several improvements have been made with respect to the previous release, especially in…
Processing high-throughput DNA sequencing data of individuals or populations requires stringing together independent software tools with many parameters, often leading to non-reproducible pipelines and datasets. We developed grenepipe to…
Scientists conduct large-scale simulations to compute derived quantities-of-interest (QoI) from primary data. Often, QoI are linked to specific features, regions, or time intervals, such that data can be adaptively reduced without…