Related papers: HDFITS: porting the FITS data model to HDF5
This work in progress paper outlines research looking at the performance impact of using different storage interfaces to access the high performance object store DAOS. We demonstrate that using DAOS through a FUSE based filesystem interface…
Scientific data is often stored in files because of the simplicity they provide in managing, transferring, and sharing data. These files are typically structured in a specific arrangement and contain metadata to understand the structure the…
The WorldWide Telescope(WWT) is a scientific visualization platform which can browse deep space images, star catalogs, and planetary remote sensing data from different observation facilities in a three-dimensional virtual scene. First…
Image subtraction is essential for transient detection in time-domain astronomy. The point spread function (PSF), photometric scaling, and sky background generally vary with time and across the field-of-view for imaging data taken with…
Exascale I/O initiatives will require new and fully integrated I/O models which are capable of providing straightforward functionality, fault tolerance and efficiency. One solution is the Distributed Asynchronous Object Storage (DAOS)…
Scientific exploitation of the ever increasing volumes of astronomical data requires efficient and practical methods for data access, visualisation, and analysis. Hierarchical sky tessellation techniques enable a multi-resolution approach…
The extensible N-Dimensional Data Format (NDF) was designed and developed in the late 1980s to provide a data model suitable for use in a variety of astronomy data processing applications supported by the UK Starlink Project. Starlink…
In the recent years, heterogeneous machine learning accelerators have become of significant interest in science, engineering and industry. The major processing speed bottlenecks in these platforms come from (a) an electronic data…
We compare a variety of lossless image compression methods on a large sample of astronomical images and show how the compression ratios and speeds of the algorithms are affected by the amount of noise in the images. In the ideal case where…
The increasing volumes of astronomical data require practical methods for data exploration, access and visualisation. The Hierarchical Progressive Survey (HiPS) is a HEALPix based scheme that enables a multi-resolution approach to astronomy…
Scientific datasets are known for their challenging storage demands and the associated processing pipelines that transform their information. Some of those processing tasks include filtering, cleansing, aggregation, normalization, and data…
Managing and analyzing pose data is a complex task, with challenges ranging from handling diverse file structures and data types to facilitating effective data manipulations such as normalization and augmentation. This paper presents…
We present a procedure for efficiently compressing astronomical radio data for high performance applications. Integrated, post-correlation data are first passed through a nearly lossless rounding step which compares the precision of the…
Materials science is undergoing profound changes due to advances in characterization instrumentation that have resulted in an explosion of data in terms of volume, velocity, variety and complexity. Harnessing these data for scientific…
The gps2gtfs package addresses a critical need for converting raw Global Positioning System (GPS) trajectory data from public transit vehicles into the widely used GTFS (General Transit Feed Specification) format. This transformation…
Access to credible sustainability information in the fashion industry remains limited and challenging to interpret, despite growing public and regulatory demands for transparency. General-purpose language models often lack domain-specific…
Understanding how galaxies form and evolve requires measuring their light distributions in images taken by telescopes. This process often involves fitting mathematical models to galaxy images to extract properties such as size, brightness,…
Archival aerial imagery is a source of worldwide very high resolution data for documenting paste 3-D changes. However, external information is required so that accurate 3-D models can be computed from archival aerial imagery. In this…
We investigate the performance of Apache Spark, a cluster computing framework, for analyzing data from future LSST-like galaxy surveys. Apache Spark attempts to address big data problems have hitherto proved successful in the industry, but…
HfS is a tool to fit the hyperfine structure of spectral lines, with multiple velocity components. The HfS_nh3 procedures included in HfS fit simultaneously the hyperfine structure of the NH$_3$ (J,K)= (1,1) and (2,2) transitions, and…