Related papers: Challenges for LSST scale data sets
All sciences, including astronomy, are now entering the era of information abundance. The exponentially increasing volume and complexity of modern data sets promises to transform the scientific practice, but also poses a number of common…
Astronomy is increasingly encountering two fundamental truths: (1) The field is faced with the task of extracting useful information from extremely large, complex, and high dimensional datasets; (2) The techniques of astroinformatics and…
(Abridged) The Large Synoptic Survey Telescope (LSST) is currently by far the most ambitious proposed ground-based optical survey. The main science themes that drive the LSST system design are Dark Energy and Matter, the Solar System…
The ongoing optical time-domain astronomy surveys are routinely reporting fifty transient candidates per night. Here, I investigate the demographics of astronomical transients and supernova classifications reported to the Transient Name…
The tens of millions of radio sources to be detected with next-generation surveys pose new challenges, quite apart from the obvious ones of processing speed and data volumes. For example, existing algorithms are inadequate for source…
The next-generation astronomy archives will cover most of the universe at fine resolution in many wavelengths. One of the first of these projects, the Sloan Digital Sky Survey (SDSS) will create a 5-wavelength catalog over 10,000 square…
Large-scale and categorical-balanced text data is essential for training effective Scene Text Recognition (STR) models, which is hard to achieve when collecting real data. Synthetic data offers a cost-effective and perfectly labeled…
Due to the complicated procedure and costly hardware, Simultaneous Localization and Mapping (SLAM) has been heavily dependent on public datasets for drill and evaluation, leading to many impressive demos and good benchmark scores. However,…
Here we review some of the main issues related to multi-wavelength source identification and characterization, with particular emphasis on the field of X-ray surveys carried out over the last years. This complex and time-consuming process…
With advancements in GPS, remote sensing, and computational simulation, an enormous volume of spatiotemporal data is being collected at an increasing speed from various application domains, spanning Earth sciences, agriculture, smart…
The analysis and an efficient scientific exploration of the Digital Palomar Observatory Sky Survey (DPOSS) represents a major technical challenge. The input data set consists of 3 Terabytes of pixel information, and contains a few billion…
The field of astronomy is starting to generate more data than can be managed, served and processed by current techniques. This paper has outlined practices for developing next-generation tools and techniques for surviving this data tsunami,…
Real time large scale streaming data pose major challenges to forecasting, in particular defying the presence of human experts to perform the corresponding analysis. We present here a class of models and methods used to develop an…
In this, the first in a series of three papers concerning the SuperCOSMOS Sky Survey (SSS), we give an introduction and user guide to the survey programme. We briefly describe other wide-field surveys and compare with our own. We give…
In preparation for cosmological analyses of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST), the LSST Dark Energy Science Collaboration (LSST DESC) has created a 300 deg$^2$ simulated survey as part of an effort called…
The massive data sets from today's particle physics experiments present a variety of challenges amenable to the tools developed by the statistics community. From the real-time decision of what subset of data to record on permanent storage,…
(Abridged) We consider the scientific case for a large aperture (10-12m class) optical spectroscopic survey telescope with a field of view comparable to that of LSST. Such a facility could enable transformational progress in several areas…
The Wide-field Spectroscopic Telescope (WST) is proposed as a new facility dedicated to the efficient delivery of spectroscopic surveys. This white paper summarises the initial concept as well as the corresponding science cases. WST will…
The advent of experimental science facilities-instruments and observatories, such as the Large Hadron Collider, the Laser Interferometer Gravitational Wave Observatory, and the upcoming Large Synoptic Survey Telescope-has brought about…
The advent of wide-area multicolour synoptic sky surveys is leading to data sets unprecedented in size, complexity and data throughput. VO technology offers a way to exploit these to the full but requires changes in design philosophy. The…