Related papers: Spatial Indexing of Large Multidimensional Databas…
Data-intensive applications often require exploratory analysis of large datasets. If analysis is performed on distributed resources, data locality can be crucial to high throughput and performance. We propose a "data diffusion" approach…
Clustering is an effective tool for astronomical spectral analysis, to mine clustering patterns among data. With the implementation of large sky surveys, many clustering methods have been applied to tackle spectroscopic and photometric data…
Astronomy produces extremely large data sets from ground-based telescopes, space missions, and simulation. The volume and complexity of these rich data sets require new approaches and advanced tools to understand the information contained…
We review the current state of data mining and machine learning in astronomy. 'Data Mining' can have a somewhat mixed connotation from the point of view of a researcher in this field. If used correctly, it can be a powerful approach,…
Time-domain astronomy (TDA) is facing a paradigm shift caused by the exponential growth of the sample size, data complexity and data generation rates of new astronomical sky surveys. For example, the Large Synoptic Survey Telescope (LSST),…
Sky coverage is one of the most important pieces of information about astronomical observations. We discuss possible representations, and present algorithms to create and manipulate shapes consisting of generalized spherical polygons with…
Perhaps one of the mostly hotly debated topics in recent years has been the question of "GIS and Big Data". Much of the discussion has been about the data: huge volumes of 2D and 3D spatial data and spatio-temporal data are now being…
Large language models (LLMs) are advancing rapidly. Such models have demonstrated strong capabilities in learning from large-scale (unstructured) text data and answering user queries. Users do not need to be experts in structured query…
We explored ways of doing spatial search within a relational database: (1) hierarchical triangular mesh (a tessellation of the sphere), (2) a zoned bucketing system, and (3) representing areas as disjunctive-normal form constraints. Each of…
Filtering data based on predicates is one of the most fundamental operations for any modern data warehouse. Techniques to accelerate the execution of filter expressions include clustered indexes, specialized sort orders (e.g., Z-order),…
The next decade will feature a growing number of massive ground-based photometric, spectroscopic, and time-domain surveys, including those produced by DECam, DESI, and LSST. The NOAO Data Lab was launched in 2017 to enable efficient…
Modern cloud databases present scaling as a binary decision: scale-out by adding nodes or scale-up by increasing per-node resources. This one-dimensional view is limiting because database performance, cost, and coordination overhead emerge…
We describe the design and implementation of a high performance cloud that we have used to archive, analyze and mine large distributed data sets. By a cloud, we mean an infrastructure that provides resources and/or services over the…
A recent research trend involves treating database index structures as Machine Learning (ML) models. In this domain, single or multiple ML models are trained to learn the mapping from keys to positions inside a data set. This class of…
In various astronomical projects it is crucial to have coordinates indexed tables. All sky optical and IR catalogues have up to 1 billion objects that will increase with forthcoming projects. Also partial sky surveys at various wavelengths…
The Sloan Digital Sky Survey (SDSS) will provide the data to support detailed investigations of the distribution of luminous and non- luminous matter in the Universe: a photometrically and astrometrically calibrated digital imaging survey…
In this paper we have focused a variety of techniques, approaches and different areas of the research which are helpful and marked as the important field of data mining Technologies. As we are aware that many Multinational companies and…
Modern astronomy relies on massive databases collected by robotic telescopes and digital sky surveys, acquiring data in a much faster pace than what manual analysis can support. Among other data, these sky surveys collect information about…
The area of scientific research that deals with the simultaneous optimization of several (possibly conflicting) criteria is named multi-objective optimization. The ability to efficiently filter and extract interesting data out of large…
Temporal data, notably time series and spatio-temporal data, are prevalent in real-world applications. They capture dynamic system measurements and are produced in vast quantities by both physical and virtual sensors. Analyzing these data…