English

scida: scalable analysis for scientific big data

Instrumentation and Methods for Astrophysics 2024-02-29 v1 Cosmology and Nongalactic Astrophysics Astrophysics of Galaxies

Abstract

scida is a Python package for reading and analyzing large scientific data sets with support for various cosmological and galaxy formation simulations out-of-the-box. Data access is provided through a hierarchical dictionary-like data structure after a simple load() function. Using the dask library for scalable, parallel and out-of-core computation, all computation requests from a user session are first collected in a task graph. Arbitrary custom analysis, as well as all available dask (array) operations, can be performed. The subsequent computation is executed only upon request, on a target resource (e.g. a HPC cluster).

Keywords

Cite

@article{arxiv.2402.17818,
  title  = {scida: scalable analysis for scientific big data},
  author = {Chris Byrohl and Dylan Nelson},
  journal= {arXiv preprint arXiv:2402.17818},
  year   = {2024}
}

Comments

recommended for acceptance in the Journal of Open Source Software; open-source development at https://github.com/cbyrohl/scida

R2 v1 2026-06-28T15:02:27.513Z