Related papers: Astronomical Pipeline Provenance: A Use Case Evalu…
To benefit from the abundance of data and the insights it brings data processing pipelines are being used in many areas of research and development in both industry and academia. One approach to automating data processing pipelines is the…
Successful data-driven science requires complex data engineering pipelines to clean, transform, and alter data in preparation for machine learning, and robust results can only be achieved when each step in the pipeline can be justified, and…
The digital revolution is transforming astronomy from a data-starved to a data-submerged science. Instruments such as the Atacama Large Millimeter Array (ALMA), the Large Synoptic Survey Telescope (LSST), and the Square Kilometer Array…
Apache Spark is a Big Data framework for working on large distributed datasets. Although widely used in the industry, it remains rather limited in the academic community or often restricted to software engineers. The goal of this paper is…
The unsurpassed sensitivity and resolution of the Square Kilometer Array (SKA) will make it possible for the first time to probe the continuum emission of normal star forming galaxies out to the edges of the universe. This opens the…
The next generation of telescopes such as the SKA and the Rubin Observatory will produce enormous data sets, requiring automated anomaly detection to enable scientific discovery. Here, we present an overview and friendly user guide to the…
The SKA Observatory (SKAO), a landmark project in radio astronomy, seeks to address fundamental questions in astronomy. To process its immense data output, approximately 700 PB/year, a global network of SKA Regional Centres (SR-CNet) will…
The upcoming Square Kilometer Array (SKA) is expected to produce humongous amount of data for undertaking H{\sc i}~science. We have developed an MPI-based {\sc Python} pipeline to deal with the large data efficiently with the present…
Object detection in astronomical images, generically referred to as source finding, is often performed before the object characterisation stage in astrophysical processing work flows. In radio astronomy, source finding has historically been…
We present the results of our investigations into options for the computing platform for the imaging pipeline in the CHILES project, an ultra-deep HI pathfinder for the era of the Square Kilometre Array. CHILES pushes the current computing…
The Square Kilometre Array is conceived as a telescope which will both test fundamental physical laws and transform our current picture of the Universe. However, the scientific challenges outlined in this book are today's problems--will…
The Square Kilometre Array (SKA, https://skatelescope.org) will be the world's largest radio telescope. SKA Science Data Challenges will be regularly issued to the community as part of the science preparatory activities. The purpose of…
We describe the software requirement and design specifications for all-sky panoramic astronomical pipelines. The described software aims to meet the specific needs of super-wide angle optics, and includes cosmic-ray hit rejection, image…
The Square Kilometre Array (SKA), when it becomes functional, is expected to enrich neutron star (NS) catalogues by at least an order of magnitude over their current state. This includes the discovery of new NS objects leading to better…
The Square Kilometre Array (SKA) is the next generation radio telescope distinguished by a superb sensitivity due to its large aperture (about one square kilometre) and advanced instrumentation. It will cover a broad range of observing…
The Square Kilometre Array (SKA) is an integral part of the next-generation observatories that will survey the Universe across the electromagnetic spectrum, and beyond, revolutionizing our view of fundamental physics, astrophysics and…
Computational workflow management systems power contemporary data-intensive sciences. The slowly resolving reproducibility crisis presents both a sobering warning and an opportunity to iterate on what science and data processing entails.…
Both Phase 1 of the Square Kilometre Array (SKA1) and the full SKA have the potential to dramatically increase the science return from future astrophysics, heliophysics, and especially planetary missions, primarily due to the greater…
The contemporary astronomical instruments have been producing the unprecedented amount of data. The largest part of this "data avalanche" is being produced by deep all-sky surveys yielding terabytes of raw data per night. Such a great data…
We review the current status of the Square Kilometre Array (SKA) by outlining the science drivers for its Phase-1 (SKA1) and setting out the timeline for the key decisions and milestones on the way to the planned start of its construction…