Related papers: Petabyte Scale Data Mining: Dream or Reality?

The next decade will feature a growing number of massive ground-based photometric, spectroscopic, and time-domain surveys, including those produced by DECam, DESI, and LSST. The NOAO Data Lab was launched in 2017 to enable efficient…

We live in a data-centric world where we are heading to generate close to 200 Zettabytes of data by the year 2025. Our data processing requirements have also increased as we push to build data processing frameworks that can process large…

In the coming decade, astronomical surveys of the sky will generate tens of terabytes of images and detect hundreds of millions of sources every night. The study of these sources will involve computation challenges such as anomaly detection…

The site conditions that make astronomical observatories in space and on the ground so desirable -- cold and dark -- demand a physical remoteness that leads to limited data transmission capabilities. Such transmission limitations directly…

It is currently feasible to start a continuous digital record of the entire sky sensitive to any visual magnitude brighter than 15 each night. Such a record could be created with a modest array of small telescopes, which collectively…

The size of astronomical observational data is increasing yearly. For example, while Atacama Large Millimeter/submillimeter Array is expected to generate 200 TB raw data every year, Large Synoptic Survey Telescope is estimated to produce 15…

Nowadays, science has been coming into a new paradigm, called data-intensive science. While current studies of the new phenomenon focused on building up infrastructure for this new paradigm, yet a few studies concern users of scientific…

Astronomical observations already produce vast amounts of data through a new generation of telescopes that cannot be analyzed manually. Next-generation telescopes such as the Large Synoptic Survey Telescope and the Square Kilometer Array…

Over the past decade, sky surveys such as the Sloan Digital Sky Survey have proven the power of large data sets for answering fundamental astrophysical questions. This observational progress, based on a synergy of advances in telescope…

Large scientific collaborations often have multiple scientists accessing the same set of files while doing different analyses, which create repeated accesses to the large amounts of shared data located far away. These data accesses have…

Astronomical datasets are growing in size and diversity, posing severe technical problems. At the same time scientific goals increasingly require the analysis of very large amounts of data, and data from multiple archives. The Virtual…

The growing resolution and volume of climate data from remote sensing and simulations pose significant storage, processing, and computational challenges. Traditional compression or subsampling methods often compromise data fidelity,…

Data collection is pervasively bound to our digital lifestyle. A recent study by the IDC reports that the growth of the data created and replicated in 2020 was even higher than in the previous years due to pandemic-related confinements to…

As observational datasets become larger and more complex, so too are the questions being asked of these data. Data simulations, i.e., synthetic data with properties (pixelization, noise, PSF, artifacts, etc.) akin to real data, are…

In the past couple of decades, the computational abilities of supercomput- ers have increased tremendously. Leadership scale supercomputers now are capable of petaflops. Likewise, the problem size targeted by applications running on such…

Data mining techniques, including clustering and classification tasks, for the automatic information extraction from large datasets are increasingly demanded in several scientific fields. In particular, in the astrophysical field, large…

A common task in scientific computing is the derivation of data. This workflow extracts the most important information from large input data and stores it in smaller derived data objects. The derived data objects can then be used for…

The Sloan Digital Sky Survey (SDSS) science database describes over 140 million objects and is over 1.5 TB in size. The SDSS Catalog Archive Server (CAS) provides several levels of query interface to the SDSS data via the SkyServer website.…

The nature of scientific and technological data collection is evolving rapidly: data volumes and rates grow exponentially, with increasing complexity and information content, and there has been a transition from static data sets to data…

The large-scale surveys such as PTF, CRTS and Pan-STARRS-1 that have emerged within the past 5 years or so employ digital databases and modern analysis tools to accentuate research into Time Domain Astronomy (TDA). Preparations are underway…