Related papers: Scaling pair count to next galaxy surveys
Exact hierarchical agglomerative clustering (HAC) of large spatial datasets is limited in practice by the $\mathcal{O}(n^2)$ time and memory required for the full pairwise distance matrix. We present GSHAC (Geographically Sparse…
We present here a new algorithm for the fast computation of N-point correlation functions in large astronomical data sets. The algorithm is based on kdtrees which are decorated with cached sufficient statistics thus allowing for orders of…
The most effective use of data from current and upcoming large scale structure~(LSS) and CMB observations requires the ability to predict the clustering of LSS with very high precision. The Effective Field Theory of Large Scale Structure…
The ability to collect unprecedented amounts of astronomical data has enabled the studying scientific questions that were impractical to study in the pre-information era. This study uses large datasets collected by four different robotic…
A fundamental procedure in the analysis of massive datasets is the construction of similarity graphs. Such graphs play a key role for many downstream tasks, including clustering, classification, graph learning, and nearest neighbor search.…
This Letter proposes a rescaled adaptive coupling scheme for the synchronization of spatially extended systems. Coupling and synchronization are analyzed from the point view of image filter construction. A length rescaling technique is…
Calculating interactions or correlations between pairs of particles is typically the most time-consuming task in particle simulation or correlation analysis. Straightforward implementations using a double loop over particle pairs have…
Spatial range joins have many applications, including geographic information systems, location-based social networking services, neuroscience, and visualization. However, joins incur not only expensive computational costs but also too large…
As dataset sizes increase, data analysis tasks in high performance computing (HPC) are increasingly dependent on sophisticated dataflows and out-of-core methods for efficient system utilization. In addition, as HPC systems grow, memory…
We present sDBSCAN, a scalable density-based clustering algorithm in high dimensions with cosine distance. Utilizing the neighborhood-preserving property of random projections, sDBSCAN can quickly identify core points and their…
We present a study on galaxy detection and shape classification using topometric clustering algorithms. We first use the DBSCAN algorithm to extract, from CCD frames, groups of adjacent pixels with significant fluxes and we then apply the…
The observation of the advancing and retreating pattern of polar sea ice cover stands as a vital indicator of global warming. This research aims to develop a robust, effective, and scalable system for classifying polar sea ice as…
A key challenge in spatial statistics is the analysis for massive spatially-referenced data sets. Such analyses often proceed from Gaussian process specifications that can produce rich and robust inference, but involve dense covariance…
This paper describes a novel approach to neighbour-finding in Smoothed Particle Hydrodynamics (SPH) simulations with large dynamic range in smoothing length. This approach is based on hierarchical cell decompositions, sorted interactions,…
To enhance the efficiency, scalability, and cross-survey applicability of stellar parameter inference in large spectroscopic datasets, we present a modular, parallelized Python framework with automated error estimation, built on the LAMOST…
Star formation is a multi-scale problem, and only global simulations that account for the connection from the molecular cloud scale gas flow to the accreting protostar can reflect the observed complexity of protostellar systems.…
Estimating stellar masses for billions of galaxies in upcoming surveys requires methods that are both accurate and computationally efficient. We present a new approach using symbolic regression trained on a simulation to derive simple,…
We present an algorithm to photometrically calibrate wide field optical imaging surveys, that simultaneously solves for the calibration parameters and relative stellar fluxes using overlapping observations. The algorithm decouples the…
For light curve generation, a pre-planned photometry survey is needed nowadays, where all of the exposure coordinates have to be given and don't change during the survey. This thesis shows it is not required and we can data-mine these light…
Graph comparison is a fundamental operation in data mining and information retrieval. Due to the combinatorial nature of graphs, it is hard to balance the expressiveness of the similarity measure and its scalability. Spectral analysis…