English

TDLight: A Framework for Incremental Light Curve Management and Smart Classification

Instrumentation and Methods for Astrophysics 2025-11-25 v1

Abstract

With the exponential growth of time-domain surveys, the volume of light curves has increased rapidly. However, many survey projects, such as Gaia, still rely on offline batch-processing workflows in which data are calibrated, merged, and released only after an observing phase is completed. This latency delays scientific analysis and causes many high-value transient events to be buried in archival data, missing the window for timely follow-up. While existing alert brokers handle heterogeneous data streams, it remains difficult to deploy a unified framework that combines high-performance incremental storage with real-time classification on local infrastructure. To address this challenge, we propose TDLight, a scalable system that adapts the time-series database TDengine (a high-performance IoT database) for astronomical data using a one-table-per-source schema. This architecture supports high-throughput ingestion, achieving 954,000 rows s^-1 for archived data and 541,000 rows s^-1 for incremental streams, while Hierarchical Equal Area isoLatitude Pixelization (HEALPix) indexing enables efficient cone-search queries. Building on this storage layer, we integrate the pre-trained hierarchical Random Forest classifier from the LEAVES framework to construct an incremental classification pipeline. Using the LEAVES dataset, we simulate data accumulation and evaluate a trigger-based strategy that performs early classification at specific observational milestones. In addition, by monitoring the evolution of classification probabilities, the system identifies "high-value candidates" -- sources that show high early confidence but later undergo significant label shifts. TDLight is released as an open-source Dockerized environment, providing a deployable infrastructure for next-generation time-domain surveys.

Keywords

Cite

@article{arxiv.2511.18784,
  title  = {TDLight: A Framework for Incremental Light Curve Management and Smart Classification},
  author = {Xinghang Yu and Ce Yu and Zeguang Shao and Bin Yang},
  journal= {arXiv preprint arXiv:2511.18784},
  year   = {2025}
}

Comments

8 pages, 6 figures

R2 v1 2026-07-01T07:51:35.210Z