English

superB/NRPy: Scalable, Task-Based Numerical Relativity for 3G Gravitational Wave Science

General Relativity and Quantum Cosmology 2025-07-29 v2 Instrumentation and Methods for Astrophysics Computational Physics

Abstract

Modern gravitational-wave science demands increasingly accurate and computationally intensive numerical relativity (NR) simulations. The Python-based, open-source NRPy framework generates optimized C/C++ code for NR, including the complete NR code BlackHoles@Home (BH@H), which leverages curvilinear coordinates well-suited to many astrophysical scenarios. Historically, BH@H was limited to single-node OpenMP CPU parallelism. To address this, we introduce superB, an open-source extension to NRPy that enables automatic generation of scalable, task-based, distributed-memory Charm++ code from existing BH@H modules. The generated code partitions the structured grids used by NRPy/BH@H, managing communication between them. Its correctness is validated through bit-identical results with the standard OpenMP version on a single node and via a head-on binary black hole simulation in cylindrical-like coordinates, accurately reproducing quasi-normal modes (up to =8\ell=8). The superB/NRPy-generated code demonstrates excellent strong scaling, achieving an 45\approx 45x speedup on 64 nodes (7168 cores) compared to the original single-node OpenMP code for a large 3D vacuum test. This scalable infrastructure benefits demanding simulations and lays the groundwork for future multi-patch grid support, targeting long inspirals, extreme parameter studies, and rapid follow-ups. This infrastructure readily integrates with other NRPy/BH@H-based projects, enabling performant scaling for the general relativistic hydrodynamics code GRoovy, and facilitating future coupling with GPU acceleration via the NRPy-CUDA project.

Cite

@article{arxiv.2505.00097,
  title  = {superB/NRPy: Scalable, Task-Based Numerical Relativity for 3G Gravitational Wave Science},
  author = {Nishita Jadoo and Terrence Pierre Jacques and Zachariah B. Etienne},
  journal= {arXiv preprint arXiv:2505.00097},
  year   = {2025}
}

Comments

24 pages, 7 figures; updated to match CQG version

R2 v1 2026-06-28T23:17:19.263Z