Related papers: Petabyte Scale Data Mining: Dream or Reality?
The changing heavens have played a central role in the scientific effort of astronomers for centuries. Galileo's synoptic observations of the moons of Jupiter and the phases of Venus starting in 1610, provided strong refutation of Ptolemaic…
The Legacy Survey of Space and Time, operated by the Vera C. Rubin Observatory, is a 10-year astronomical survey due to start operations in 2022 that will image half the sky every three nights. LSST will produce ~20TB of raw data per night…
The abundance of available sensor and derived data from large scientific experiments, such as earth observation programs, radio astronomy sky surveys, and high-energy physics already exceeds the storage hardware globally fabricated per…
Over the past century, major advances in astronomy and astrophysics have been largely driven by improvements in instrumentation and data collection. With the amassing of high quality data from new telescopes, and especially with the advent…
Most scientific data will never be directly examined by scientists; rather it will be put into online databases where it will be analyzed and summarized by computer programs. Scientists increasingly see their instruments through online…
We describe DPOSS, a new digital survey of the northern sky, based on the POSS-II photographic sky atlas. The survey covers the entire sky north of delta = -3 deg in 3 bands, calibrated to the Gunn $gri$ system, reaching to equivalent…
Despite centuries of close association, statistics and astronomy are surprisingly distant today. Most observational astronomical research relies on an inadequate toolbox of methodological tools. Yet the needs are substantial: astronomy…
In this decade astronomy is undergoing a paradigm shift to handle data from next generation observatories such as the Square Kilometre Array (SKA) or the Vera C. Rubin Observatory (LSST). Producing real time data streams of up to 10 TB/s…
The goal of the Sloan Digital Sky Survey is ``to map in detail one-quarter of the entire sky, determining the positions and absolute brightnesses of more than 100 million celestial objects''. The survey will be performed by taking…
Cloud computing offers an opportunity to run compute-resource intensive climate models at scale by parallelising model runs such that datasets useful to the exoplanet community can be produced efficiently. To better understand the…
Scientists generate petabytes of data daily to help uncover environmental trends or behaviors that are hard to predict. For example, understanding climate simulations based on the long-term average of temperature, precipitation, and other…
Introduction: Before embarking on the design of any computer system it is first necessary to assess the magnitude of the problem. In the case of a web search engine this assessment amounts to determining the current size of the web, the…
The excessive amounts of data generated by devices and Internet-based sources at a regular basis constitute, big data. This data can be processed and analyzed to develop useful applications for specific domains. Several mathematical and…
The SKA will be capable of producing a stream of science data products that are Exa-scale in terms of their storage and processing requirements. This Google-scale enterprise is attracting considerable international interest and excitement…
Data challenges are emerging as powerful tools with which to answer fundamental astronomical questions. Time-domain astronomy lends itself to data challenges, particularly in the era of classification and anomaly detection. With improved…
As the amount of scientific data continues to grow at ever faster rates, the research community is increasingly in need of flexible computational infrastructure that can support the entirety of the data science lifecycle, including…
The contemporary astronomical instruments have been producing the unprecedented amount of data. The largest part of this "data avalanche" is being produced by deep all-sky surveys yielding terabytes of raw data per night. Such a great data…
It is argued that the astronomy of the twenty-first century will be dominated by computer-based manipulation of huge homogeneous surveys of various types of astronomical objects. Furthermore combination of all observations with large…
The next generation of telescopes and instruments are facilitating our understanding of the Universe by producing data at a pace that beats all projections, and astronomers today are left in the face of an avalanche of data like never…
Astronomy is changing. Large projects, large collaborations, and large budgets are becoming the norm. The Sloan Digital Sky Survey (SDSS) is one example of this new astronomy, and in operating the original survey, we put in place and…