Related papers: Bifrost: a Python/C++ Framework for High-Throughpu…
The Pierre Auger Observatory is designed to unveil the nature and the origins of the highest energy cosmic rays. The large and geographically dispersed collaboration of physicists and the wide-ranging collection of simulation and…
The process monitoring task is characterized by stringent demands for accuracy and efficiency. Current transformer-based methods, characterized by self-attention for temporal fusion, exhibit limitations in accurately understanding the…
The Large Synoptic Survey Telescope (LSST) is an ambitious astronomical survey with a similarly ambitious Data Management component. Data Management for LSST includes processing on both nightly and yearly cadences to generate transient…
PaPy, which stands for parallel pipelines in Python, is a highly flexible framework that enables the construction of robust, scalable workflows for either generating or processing voluminous datasets. A workflow is created from user-written…
Collaborations in astronomy and astrophysics are faced with numerous cyber infrastructure challenges, such as large data sets, the need to combine heterogeneous data sets, and the challenge to effectively collaborate on those large,…
Effective performance profiling and analysis are essential for optimizing training and inference of deep learning models, especially given the growing complexity of heterogeneous computing environments. However, existing tools often lack…
We describe a scalable distributed imaging algorithm framework for next-generation radio telescopes, managing the Fourier transform from apertures to sky (or vice versa) with a focus on minimising memory load, data transfers, and…
As enterprises move to a cloud-first approach, their network becomes crucial to their daily operations and has to be continuously monitored. Although passive monitoring can be convenient from a deployment viewpoint, inferring the state of…
Petabytes of archival high time resolution observations have been captured with the Murchison Widefield Array. The search for Fast Radio Bursts within these using established software has been limited by its inability to scale on…
The connect-fill-run workflow paradigm, widely adopted in mature software engineering, accelerates collaborative development. However, computational chemistry, computational materials science, and computational biology face persistent…
Scientific communities naturally tend to organize around data ecosystems created by the combination of their observational devices, their data repositories, and the workflows essential to carry their research from observation to discovery.…
We introduce a Python framework designed to automate the most common tasks associated with the extraction and upscaling of the statistics of single-impact crater functions to inform coefficients of continuum equations describing surface…
Over the last two decades, scientific workflow management systems (SWfMS) have emerged as a means to facilitate the design, execution, and monitoring of reusable scientific data processing pipelines. At the same time, the amounts of data…
In this thesis we investigate high throughput computational methods for processing large quantities of data collected from synchrotrons and their application to spectral analysis of powder diffraction data. We also present the main product…
Breadth-First Search (BFS) is a building block used in a wide array of graph analytics and is used in various network analysis domains: social, road, transportation, communication, and much more. Over the last two decades, network sizes…
Modern radio telescopes generate large amounts of data, with the next generation Very Large Array (ngVLA) and the Square Kilometre Array (SKA) expected to feed up to 292 GB of visibilities per second to the science data processor (SDP).…
Data preprocessing is a fundamental part of any machine learning application and frequently the most time-consuming aspect when developing a machine learning solution. Preprocessing for deep learning is characterized by pipelines that…
An essential part of building a data-driven organization is the ability to handle and process continuous streams of data to discover actionable insights. The explosive growth of interconnected devices and the social Web has led to a large…
A fast-turnaround pipeline for realtime data reduction plays an essential role in discovering and permitting follow-up observations to young supernovae and fast-evolving transients in modern time-domain surveys. In this paper, we present…
The rapidly growing demand for high-quality data in Large Language Models (LLMs) has intensified the need for scalable, reliable, and semantically rich data preparation pipelines. However, current practices remain dominated by ad-hoc…