Related papers: Lessons Learned from Sloan Digital Sky Survey Oper…
As our capacity to study ever-expanding domains of our science has increased (including the time domain, non-electromagnetic phenomena, magnetized plasmas, and numerous sky surveys in multiple wavebands with broad spatial coverage and…
I review the status of science with wide field surveys. For many decades surveys have been the backbone of astronomy, and the main engine of discovery, as we have mapped the sky at every possible wavelength. Surveys are an efficient use of…
Astronomy is experiencing a rapid growth in data size and complexity. This change fosters the development of data-driven science as a useful companion to the common model-driven data analysis paradigm, where astronomers develop automatic…
Sky surveys are the largest data generators in astronomy, making automated tools for extracting meaningful scientific information an absolute necessity. We show that, without the need for labels, self-supervised learning recovers…
Data volumes from multiple sky surveys have grown from gigabytes into terabytes during the past decade, and will grow from terabytes into tens (or hundreds) of petabytes in the next decade. This exponential growth of new data both enables…
An array of large observational programs using ground-based and space-borne telescopes is planned in the next decade. The forthcoming wide-field sky surveys are expected to deliver a sheer volume of data exceeding an exabyte. Processing the…
Most major discoveries in astronomy are unplanned, and result from surveying the Universe in a new way, rather than by testing a hypothesis or conducting an investigation with planned outcomes. For example, of the 10 greatest discoveries…
Context: Surveys constitute an valuable tool to capture a large-scale snapshot of the state of the practice. Apparently trivial to adopt, surveys hide, however, several pitfalls that might hinder rendering the result valid and, thus,…
We review some aspects of the current state of data-intensive astronomy, its methods, and some outstanding data analysis challenges. Astronomy is at the forefront of "big data" science, with exponentially growing data volumes and data…
The Sloan Digital Sky Survey science database is approaching 2TB. While the vast majority of queries normally execute in seconds or minutes, this interactive execution time can be disproportionately increased by a small fraction of queries…
We describe the application of data mining algorithms to research problems in astronomy. We posit that data mining has always been fundamental to astronomical research, since data mining is the basis of evidence-based discovery, including…
Galaxy Surveys are enjoying a renaissance thanks to the advent of multi-object spectrographs on ground-based telescopes. The last 15 years have seen the fruits of this experimental advance, including the 2-degree Field Galaxy Redshift…
Onboard autonomy technologies such as planning and scheduling, identification of scientific targets, and content-based data summarization, will lead to exciting new space science missions. However, the challenge of operating missions with…
Astronomy, as a field, has long encouraged the development of free, open digital infrastructure (e.g., National Research Council 2010, 2011). Examples range from simple scripts that enable individual scientific research, through software…
Astronomy has a distinguished tradition of using technology to accelerate the quality and effectiveness of science, and data-intensive initiatives such as the Virtual Observatory lead the way amongst other fields of science. However,…
Astronomy is one of the most data-intensive of the sciences. Data technology is accelerating the quality and effectiveness of its research, and the rate of astronomical discovery is higher than ever. As a result, many view astronomy as…
This year marks the 20th anniversary of the effective beginning of large, systematic redshift surveys of galaxies. These surveys have had a major impact on observational cosmology and on our current understanding of large-scale structures…
An earlier paper (Szalay et. al. "Designing and Mining MultiTerabyte Astronomy Archives: The Sloan Digital Sky Survey," ACM SIGMOD 2000) described the Sloan Digital Sky Survey's (SDSS) data management needs by defining twenty database…
Fundamental coding and software development skills are increasingly necessary for success in nearly every aspect of astronomical and astrophysical research as large surveys and high resolution simulations become the norm. However,…
Computational thinking is a key skill for space science graduates, who must apply advanced problem-solving skills to model complex systems, analyse big data sets, and develop control software for mission-critical space systems. We describe…