The IceProd Framework: Distributed Data Processing for the IceCube Neutrino Observatory
Distributed, Parallel, and Cluster Computing
2016-02-23 v3
Abstract
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, iden- tify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. IceProd is a distributed management system based on Python, XML-RPC and GridFTP. It is driven by a central database in order to coordinate and admin- ister production of simulations and processing of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, Condor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.
Keywords
Cite
@article{arxiv.1311.5904,
title = {The IceProd Framework: Distributed Data Processing for the IceCube Neutrino Observatory},
author = {M. G. Aartsen and R. Abbasi and M. Ackermann and J. Adams and J. A. Aguilar and M. Ahlers and D. Altmann and C. Arguelles and J. Auffenberg and X. Bai and M. Baker and S. W. Barwick and V. Baum and R. Bay and J. J. Beatty and J. Becker Tjus and K. -H. Becker and S. BenZvi and P. Berghaus and D. Berley and E. Bernardini and A. Bernhard and D. Z. Besson and G. Binder and D. Bindig and M. Bissok and E. Blaufuss and J. Blumenthal and D. J. Boersma and C. Bohm and D. Bose and S. Böser and O. Botner and L. Brayeur and H. -P. Bretz and A. M. Brown and R. Bruijn and J. Casey and M. Casier and D. Chirkin and A. Christov and B. Christy and K. Clark and L. Classen and F. Clevermann and S. Coenders and S. Cohen and D. F. Cowen and A. H. Cruz Silva and M. Danninger and J. Daughhetee and J. C. Davis and M. Day and C. De Clercq and S. De Ridder and P. Desiati and K. D. de Vries and M. de With and T. DeYoung and J. C. Díaz-Vélez and M. Dunkman and R. Eagan and B. Eberhardt and B. Eichmann and J. Eisch and S. Euler and P. A. Evenson and O. Fadiran and A. R. Fazely and A. Fedynitch and J. Feintzeig and T. Feusels and K. Filimonov and C. Finley and T. Fischer-Wasels and S. Flis and A. Franckowiak and K. Frantzen and T. Fuchs and T. K. Gaisser and J. Gallagher and L. Gerhardt and L. Gladstone and T. Glüsenkamp and A. Goldschmidt and G. Golup and J. G. Gonzalez and J. A. Goodman and D. Góra and D. T. Grandmont and D. Grant and P. Gretskov and J. C. Groh and A. Groß and C. Ha and A. Haj Ismail and P. Hallen and A. Hallgren and F. Halzen and K. Hanson and D. Hebecker and D. Heereman and D. Heinen and K. Helbing and R. Hellauer and S. Hickford and G. C. Hill and K. D. Hoffman and R. Hoffmann and A. Homeier and K. Hoshina and F. Huang and W. Huelsnitz and P. O. Hulth and K. Hultqvist and S. Hussain and A. Ishihara and E. Jacobi and J. Jacobsen and K. Jagielski and G. S. Japaridze and K. Jero and O. Jlelati and B. Kaminsky and A. Kappes and T. Karg and A. Karle and M. Kauer and J. L. Kelley and J. Kiryluk and J. Kläs and S. R. Klein and J. -H. Köhne and G. Kohnen and H. Kolanoski and L. Köpke and C. Kopper and S. Kopper and D. J. Koskinen and M. Kowalski and M. Krasberg and A. Kriesten and K. Krings and G. Kroll and J. Kunnen and N. Kurahashi and T. Kuwabara and M. Labare and H. Landsman and M. J. Larson and M. Lesiak-Bzdak and M. Leuermann and J. Leute and J. Lünemann and O. Macías and J. Madsen and G. Maggi and R. Maruyama and K. Mase and H. S. Matis and F. McNally and K. Meagher and M. Merck and G. Merino and T. Meures and S. Miarecki and E. Middell and N. Milke and J. Miller and L. Mohrmann and T. Montaruli and R. Morse and R. Nahnhauer and U. Naumann and H. Niederhausen and S. C. Nowicki and D. R. Nygren and A. Obertacke and S. Odrowski and A. Olivas and A. Omairat and A. O'Murchadha and L. Paul and J. A. Pepper and C. Pérez de los Heros and C. Pfendner and D. Pieloth and E. Pinat and J. Posselt and P. B. Price and G. T. Przybylski and M. Quinnan and L. R ädel and I. Rae and M. Rameez and K. Rawlins and P. Redl and R. Reimann and E. Resconi and W. Rhode and M. Ribordy and M. Richman and B. Riedel and J. P. Rodrigues and C. Rott and T. Ruhe and B. Ruzybayev and D. Ryckbosch and S. M. Saba and H. -G. Sander and M. Santander and S. Sarkar and K. Schatto and F. Scheriau and T. Schmidt and M. Schmitz and S. Schoenen and S. Schöneberg and A. Schönwald and A. Schukraft and L. Schulte and D. Schultz and O. Schulz and D. Seckel and Y. Sestayo and S. Seunarine and R. Shanidze and C. Sheremata and M. W. E. Smith and D. Soldin and G. M. Spiczak and C. Spiering and M. Stamatikos and T. Stanev and N. A. Stanisha and A. Stasik and T. Stezelberger and R. G. Stokstad and A. Stößl and E. A. Strahler and R. Ström and N. L. Strotjohann and G. W. Sullivan and H. Taavola and I. Taboada and A. Tamburro and A. Tepe and S. Ter-Antonyan and G. Tešić and S. Tilav and P. A. Toale and M. N. Tobin and S. Toscano and M. Tselengidou and E. Unger and M. Usner and S. Vallecorsa and N. van Eijndhoven and A. Van Overloop and J. van Santen and M. Vehring and M. Voge and M. Vraeghe and C. Walck and T. Waldenmaier and M. Wallraff and Ch. Weaver and M. Wellons and C. Wendt and S. Westerhoff and N. Whitehorn and K. Wiebe and C. H. Wiebusch and D. R. Williams and H. Wissing and M. Wolf and T. R. Wood and K. Woschnagg and D. L. Xu and X. W. Xu and J. P. Yanez and G. Yodh and S. Yoshida and P. Zarzhitsky and J. Ziemann and S. Zierke and M. Zoll},
journal= {arXiv preprint arXiv:1311.5904},
year = {2016}
}