A Highly Efficient Cross-matching Scheme using Learned Index Structure
Abstract
Spatial data fusion is a bottleneck when it meets the scale of 10 billion records. Cross-matching celestial catalogs is just one example of this. To challenge this, we present a framework that enables efficient cross-matching using Learned Index Structures. Our approach involves a data transformation method to map multi-dimensional data into easily learnable distributions, coupled with a novel search algorithm that leverages the advantages of model pairs, significantly enhancing the efficiency of nearest-neighbor search. In this study, we utilized celestial catalog data derived from astronomical surveys to construct the index and evaluated the speed of the cross-matching process. Using the HEALPix segmentation scheme, we built an independent model object for each tile and developed an end-to-end pipeline to construct a framework with semantic guarantees for record retrieval in query and range search. Our results show that the proposed method improves cross-matching speed by more than four times compared to KD-trees for a radius range between 1 milli-arcseconds and 100 arcseconds.
Cite
@article{arxiv.2504.10931,
title = {A Highly Efficient Cross-matching Scheme using Learned Index Structure},
author = {Phu-Minh Lam and Dongwei Fan and Hongbo Wei and Jun Wang and Yu Zhou and Qi Ma and Baolong Zhang and Xiazhao Zhang and Yongheng Wang},
journal= {arXiv preprint arXiv:2504.10931},
year = {2025}
}