Related papers: The Data Processing Pipeline for the Herschel-HIFI…
In this paper, we describe the optical imaging data processing pipeline developed for the Subaru Telescope's Hyper Suprime-Cam (HSC) instrument. The HSC Pipeline builds on the prototype pipeline being developed by the Large Synoptic Survey…
The MIMIC-IV dataset is a large, publicly available electronic health record (EHR) resource widely used for clinical machine learning research. It comprises multiple modalities, including structured data, clinical notes, waveforms, and…
Context. In the current era of multi-wavelength and multi-messenger astronomy, international organisations are actively working on the definition of new standards for the publication of astronomical data, and substantial effort is devoted…
A program that we call the QUEST Data Processing Software Pipeline has been written to process the large volumes of data produced by the QUEST camera on the Samuel Oschin Schmidt Telescope at the Palomar Observatory. The program carries out…
The increasing adoption of low-cost environmental sensors and AI-enabled applications has accelerated the demand for scalable and resilient data infrastructures, particularly in data-scarce and resource-constrained regions. This paper…
The CIWS-FW is aimed at providing a common and standard solution for the storage, processing and quick look at the data acquired from scientific instruments for astrophysics. The target system is the instrument workstation either in the…
ScopeSim is a flexible multipurpose instrument data simulation framework built in Python. It enables both raw and reduced observation data to be simulated for a wide range of telescopes and instruments quickly and efficiently on a personal…
We describe the spectroscopic data processing pipeline of the Dark Energy Spectroscopic Instrument (DESI), which is conducting a redshift survey of about 40 million galaxies and quasars using a purpose-built instrument on the 4-m Mayall…
Smart city management is going through a remarkable transition, in terms of quality and diversity of services provided to the end-users. The stakeholders that deliver pervasive applications are now able to address fundamental challenges in…
As several large single-dish radio surveys begin operation within the coming decade, a wealth of radio data will become available and provide a new window to the Universe. In order to fully exploit the potential of these data sets, it is…
Progress in science is deeply bound to the effective use of high-performance computing infrastructures and to the efficient extraction of knowledge from vast amounts of data. Such data comes from different sources that follow a cycle…
Observational astronomy has changed drastically in the last decade: manually driven target-by-target instruments have been replaced by fully automated robotic telescopes. Data acquisition methods have advanced to the point that terabytes of…
As software systems increase in complexity, conventional monitoring methods struggle to provide a comprehensive overview or identify performance issues, often missing unexpected problems. Observability, however, offers a holistic approach,…
Processing of raw data from modern astronomical instruments is nowadays often carried out using dedicated software, so-called "pipelines" which are largely run in automated operation. In this paper we describe the data reduction pipeline of…
High performance computing has been used in various fields of astrophysical research. But most of it is implemented on massively parallel systems (supercomputers) or graphical processing unit clusters. With the advent of multicore…
Digital Access to a Sky Century @ Harvard (DASCH) is a project to digitize the collection of approximately 525,000 astronomical plates held at the Harvard College Observatory. This paper presents an overview of the DASCH data processing…
Very High Resolution satellite and aerial imagery are used to monitor and conduct large scale surveys of ecological systems. Convolutional Neural Networks have successfully been employed to analyze such imagery to detect large animals and…
Software Process Improvement (SPI) encompasses the analysis and modification of the processes within software development, aimed at improving key areas that contribute to the organizations' goals. The task of evaluating whether the selected…
As spatial and temporal resolutions of scientific instruments improve, the explosion in the volume of data produced is becoming a key challenge. It can be a critical bottleneck for integration between scientific instruments at the edge and…
We develop the Mechanic package, which is a new numerical framework for dynamical astronomy. The aim of our software is to help in massive numerical simulations by efficient task management and unified data storage. The code is built on top…