Related papers: Pipeline Collector: gathering performance data for…
Performance of the Level-2 pipeline, which translates the UVIT data created by the ISRO's ground segment processing systems (Level-1) into astronomer ready scientific data products, is described. This pipeline has evolved significantly from…
Radio astronomy relies heavily on efficient and accurate processing pipelines to deliver science ready data. With the increasing data flow of modern radio telescopes, manual configuration of such data processing pipelines is infeasible.…
The Murchison Widefield Array (MWA) is a next-generation radio telescope currently under construction in the remote Western Australia Outback. Raw data will be generated continuously at 5GiB/s, grouped into 8s cadences. This high throughput…
KMOS is a multi-object near-infrared integral field spectrometer with 24 deployable pick-off arms. Data processing is inevitably complex. We discuss specific issues and requirements that must be addressed in the data reduction pipeline, the…
The VST Telescope Control Software logs continuously detailed information about the telescope and instrument operations. Commands, telemetries, errors, weather conditions and anything may be relevant for the instrument maintenance and the…
Scientists are drawn to synchrotrons and accelerator based light sources because of their brightness, coherence and flux. The rate of improvement in brightness and detector technology has outpaced Moore's law growth seen for computers,…
The evolution of distributed architectures and programming paradigms for performance-oriented program development, challenge the state-of-the-art technology for performance tools. The area of high performance computing is rapidly expanding…
As part of an on-going effort to simplify the data analysis path for VLBI experiments, a pipeline procedure has been developed at JIVE to carry out much of the data reduction required for EVN experiments in an automated fashion. This…
We present the current status of two new fully automated reduction and analysis pipelines, built for the Euler telescope and the CORALIE spectrograph. Both pipelines have been designed and built independently at the Universidad de Chile and…
Modern radio interferometers such as the LOw Frequency ARray (LOFAR) are capable of producing data at hundreds of gigabits to terabits per second. This high data rate makes the analysis of radio data cumbersome and computationally…
Modern high-sensitivity radio telescopes are discovering an increased number of resolved sources with intricate radio structures and fainter radio emissions. These sources often present a challenge because source detectors might identify…
Particle filters are computational techniques for estimating the state of dynamical systems by integrating observational data with model predictions. This work introduces a class of Localized Particle Filters (LPFs) that exploit spatial…
The Stratospheric Observatory for Infrared Astronomy (SOFIA) is an airborne astronomical observatory comprised of a 2.5-meter telescope mounted in the aft section of a Boeing 747SP aircraft. During routine operations, several instruments…
At the Canadian Astronomy Data Centre, we have combined our cloud computing system, CANFAR, with the world's most advanced machine learning software, Skytree, to create the world's first cloud computing system for data mining in astronomy.…
We anticipate that the data acquired by the Laser Interferometer Space Antenna (LISA) will be dominated by the gravitational wave signals from several astrophysical populations. The analysis of these data is a new challenge and is the main…
The recent explosion of recorded digital data and its processed derivatives threatens to overwhelm researchers when analysing their experimental data or when looking up data items in archives and file systems. While current hardware…
Large-scale distributed computing systems often contain thousands of distributed nodes (machines). Monitoring the conditions of these nodes is important for system management purposes, which, however, can be extremely resource demanding as…
The input data pipeline is an essential component of each machine learning (ML) training job. It is responsible for reading massive amounts of training data, processing batches of samples using complex transformations, and loading them onto…
In this paper, we describe the optical imaging data processing pipeline developed for the Subaru Telescope's Hyper Suprime-Cam (HSC) instrument. The HSC Pipeline builds on the prototype pipeline being developed by the Large Synoptic Survey…
We present a new pipeline utilizing machine learning for classifying short-duration features in raw time-ordered data (TOD) of cosmic microwave background survey observations. The pipeline, specifically designed for the Atacama Cosmology…