Related papers: Building a scalable global data processing pipelin…
In the coming decade, astronomical surveys of the sky will generate tens of terabytes of images and detect hundreds of millions of sources every night. The study of these sources will involve computation challenges such as anomaly detection…
Upcoming and future astronomy research facilities will systematically generate terabyte-sized data sets moving astronomy into the Petascale data era. While such facilities will provide astronomers with unprecedented levels of accuracy and…
Coming high-cadence wide-field optical telescopes will image hundreds of thousands of sources per minute. Besides inspecting the near real-time data streams for transient and variability events, the accumulated data archive is a wealthy…
The volume of data generated by modern astronomical telescopes is extremely large and rapidly growing. However, current high-performance data processing architectures/frameworks are not well suited for astronomers because of their…
The Pan-STARRS Data Processing System is responsible for the steps needed to downloaded, archive, and process all images obtained by the Pan-STARRS telescopes, including real-time detection of transient sources such as supernovae and moving…
The site conditions that make astronomical observatories in space and on the ground so desirable -- cold and dark -- demand a physical remoteness that leads to limited data transmission capabilities. Such transmission limitations directly…
This paper describes the MegaPipe image processing pipeline at the Canadian Astronomical Data Centre. The pipeline combines multiple images from the MegaCam mosaic camera on CFHT and combines them into a single output image. MegaPipe takes…
Modern astronomical data processing requires complex software pipelines to process ever growing datasets. For radio astronomy, these pipelines have become so large that they need to be distributed across a computational cluster. This makes…
The primary task of the 1.26-m telescope jointly operated by the National Astronomical Observatory and Guangzhou University is photometric observations of the g, r, and i bands. A data processing pipeline system was set up with mature…
Photometry of galaxies has typically focused on small, faint systems due to their interest for cosmological studies. Large angular size galaxies, on the other hand, offer a more detailed view into the properties of galaxies, but bring a…
Curating, processing, and combining large-scale medical imaging datasets from national studies is a non-trivial task due to the intense computation and data throughput required, variability of acquired data, and associated financial…
Seismology has entered the petabyte era, driven by decades of continuous recordings of broadband networks, the increase in nodal seismic experiments, and the recent emergence of Distributed Acoustic Sensing (DAS). This review explains how…
We present a case study of a cloud-based computational workflow for processing large astronomical data sets from the Murchison Widefield Array (MWA) cosmology experiment. Cloud computing is well-suited to large-scale, episodic computation…
High-dimensional astronomical data cubes provide a wealth of spectral and structural information that can be used to study astrophysical and chemical processes. The complexity and sheer size of these datasets pose significant challenges in…
The Argus Optical Array is a synoptic survey observatory, currently in development, that will have a total collecting area equivalent to a 5-meter monolithic telescope and an all-sky field of view, multiplexed from 900 commercial…
Very High Resolution satellite and aerial imagery are used to monitor and conduct large scale surveys of ecological systems. Convolutional Neural Networks have successfully been employed to analyze such imagery to detect large animals and…
We have constructed a new, fast, robust and reliable pipeline to detect variable stars from the ROTSE-IIId archival data. Turkish share of ROTSE-III archive contains approximately one million objects from a large field of view (1.85\dgr)…
Clustering is an effective tool for astronomical spectral analysis, to mine clustering patterns among data. With the implementation of large sky surveys, many clustering methods have been applied to tackle spectroscopic and photometric data…
Applications requiring real-time processing of large volumes of data have been the main driver for rethinking the traditional cloud, giving rise to novel cloud models. Distributed cloud (DC) is a model that allows users to dynamically…
The Cosmology Data Management system (CosmoDM) is an automated and flexible data management system for the processing and calibration of data from optical photometric surveys. It is designed to run on supercomputers and to minimize disk I/O…