Related papers: A Processing Pipeline for High Volume Pulsar Data …

Powerful detectors at modern experimental facilities routinely collect data at multiple GB/s. Online analysis methods are needed to enable the collection of only interesting subsets of such massive data streams, such as by explicitly…

Exploration of the time-domain radio sky has huge potential for advancing our knowledge of the dynamic universe. Past surveys have discovered large numbers of pulsars, rotating radio transients and other transient radio phenomena; however,…

The Square Kilometre Array (SKA) will be both the largest radio telescope ever constructed and the largest Big Data project in the known Universe. The first phase of the project will generate on the order of 5 zettabytes of data per year. A…

Increasingly larger number of software systems today are including data science components for descriptive, predictive, and prescriptive analytics. The collection of data science stages from acquisition, to cleaning/curation, to modeling,…

The SKA pulsar search pipeline will be used for real time detection of pulsars. Modern radio telescopes such as SKA will be generating petabytes of data in their full scale of operation. Hence experience-based and data-driven algorithms…

The data produced by the future space-based millihertz gravitational-wave detector LISA will require nontrivial pre-processing, which might affect the science results. It is crucial to demonstrate the feasibility of such processing…

The Australian Square Kilometre Array Pathfinder (ASKAP) collects images of the sky at radio wavelengths with an unprecedented field of view, combined with a high angular resolution and sub-millijansky sensitivities. The large quantity of…

The Square Kilometre Array (SKA) is a planned multi purpose radio telescope with a collecting area approaching 1 million square metres. One of the key science objectives of the SKA is to provide exquisite strong-field tests of gravitational…

Creating high-quality, large-scale datasets for large language models (LLMs) often relies on resource-intensive, GPU-accelerated models for quality filtering, making the process time-consuming and costly. This dependence on GPUs limits…

Systematic reviews, which entail the extraction of data from large numbers of scientific documents, are an ideal avenue for the application of machine learning. They are vital to many fields of science and philanthropy, but are very…

Proper quality control (QC) is time consuming when working with large-scale medical imaging datasets, yet necessary, as poor-quality data can lead to erroneous conclusions or poorly trained machine learning models. Most efforts to reduce…

New developments in data processing and visualization are being made in preparation for upcoming radioastronomical surveys planned with the Square Kilometre Array (SKA) and its precursors. A major goal is enabling extraction of science…

Data is a valuable asset, and sharing it as a product across organizations is key to building comprehensive and useful insights in fields such as science and industry. Before sharing, data often requires transformation to comply with…

Curating, processing, and combining large-scale medical imaging datasets from national studies is a non-trivial task due to the intense computation and data throughput required, variability of acquired data, and associated financial…

The Square Kilometre Array (SKA) will make ground breaking discoveries in pulsar science. In this chapter we outline the SKA surveys for new pulsars, as well as how we will perform the necessary follow-up timing observations. The SKA's wide…

Modern radio telescopes generate large amounts of data, with the next generation Very Large Array (ngVLA) and the Square Kilometre Array (SKA) expected to feed up to 292 GB of visibilities per second to the science data processor (SDP).…

The upcoming Square Kilometer Array (SKA) is expected to produce humongous amount of data for undertaking H{\sc i}~science. We have developed an MPI-based {\sc Python} pipeline to deal with the large data efficiently with the present…

With the advancement of technology, the data generated in our lives is getting faster and faster, and the amount of data that various applications need to process becomes extremely huge. Therefore, we need to put more effort into analyzing…

The analysis of data from x-ray microcalorimeters requires great care; their excellent intrinsic energy resolution cannot usually be achieved in practice without a statistically near-optimal pulse analysis and corrections for important…

The analysis and an efficient scientific exploration of the Digital Palomar Observatory Sky Survey (DPOSS) represents a major technical challenge. The input data set consists of 3 Terabytes of pixel information, and contains a few billion…