A Real Time Processing System for Big Data in Astronomy: Applications to HERA
Abstract
As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.
Cite
@article{arxiv.2104.03990,
title = {A Real Time Processing System for Big Data in Astronomy: Applications to HERA},
author = {Paul La Plante and Peter K. G. Williams and Matthew Kolopanis and Joshua S. Dillon and Adam P. Beardsley and Nicholas S. Kern and Michael Wilensky and Zaki S. Ali and Zara Abdurashidova and James E. Aguirre and Paul Alexander and Yanga Balfour and Gianni Bernardi and Tashalee S. Billings and Judd D. Bowman and Richard F. Bradley and Phil Bull and Jacob Burba and Steve Carey and Chris L. Carilli and Carina Cheng and David R. DeBoer and Matt Dexter and Eloy de Lera Acedo and John Ely and Aaron Ewall-Wice and Nicolas Fagnoni and Randall Fritz and Steven R. Furlanetto and Kingsley Gale-Sides and Brian Glendenning and Deepthi Gorthi and Bradley Greig and Jaspar Grobbelaar and Ziyaad Halday and Bryna J. Hazelton and Jacqueline N. Hewitt and Jack Hickish and Daniel C. Jacobs and Austin Julius and Joshua Kerrigan and Piyanat Kittiwisit and Saul A. Kohn and Adam Lanman and Telalo Lekalake and David Lewis and Adrian Liu and David MacMahon and Lourence Malan and Cresshim Malgas and Matthys Maree and Zachary E. Martinot and Eunice Matsetela and Andrei Mesinger and Mathakane Molewa and Miguel F. Morales and Tshegofalang Mosiane and Steven Murray and Abraham R. Neben and Bojan Nikolic and Aaron R. Parsons and Robert Pascua and Nipanjana Patra and Samantha Pieterse and Jonathan C. Pober and Nima Razavi-Ghods and Jon Ringuette and James Robnett and Kathryn Rosie and Mario G. Santos and Peter Sims and Craig Smith and Angelo Syce and Nithyanandan Thyagarajan and Haoxuan Zheng},
journal= {arXiv preprint arXiv:2104.03990},
year = {2021}
}
Comments
15 pages, 3 figures, published in Astronomy and Computing