English

Architecture of processing and analysis system for big astronomical data

Instrumentation and Methods for Astrophysics 2017-04-03 v1 Distributed, Parallel, and Cluster Computing

Abstract

This work explores the use of big data technologies deployed in the cloud for processing of astronomical data. We have applied Hadoop and Spark to the task of co-adding astronomical images. We compared the overhead and execution time of these frameworks. We conclude that performance of both frameworks is generally on par. The Spark API is more flexible, which allows one to easily construct astronomical data processing pipelines.

Keywords

Cite

@article{arxiv.1703.10979,
  title  = {Architecture of processing and analysis system for big astronomical data},
  author = {Ivan Kolosov and Sergey Gerasimov and Alexander Meshcheryakov},
  journal= {arXiv preprint arXiv:1703.10979},
  year   = {2017}
}

Comments

4 pages, to appear in the Proceedings of ADASS 2016, Astronomical Society of the Pacific (ASP) Conference Series

R2 v1 2026-06-22T19:03:55.427Z