English

Testing Identity of Distributions under Kolmogorov Distance in Polylogarithmic Space

Data Structures and Algorithms 2024-10-30 v1 Information Retrieval

Abstract

Suppose we have a sample from a distribution DD and we want to test whether D=DD = D^* for a fixed distribution DD^*. Specifically, we want to reject with constant probability, if the distance of DD from DD^* is ε\geq \varepsilon in a given metric. In the case of continuous distributions, this has been studied thoroughly in the statistics literature. Namely, for the well-studied Kolmogorov metric a test is known that uses the optimal O(1/ε2)O(1/\varepsilon^2) samples. However, this test naively uses also space O(1/ε2)O(1/\varepsilon^2), and previous work improved this to O(1/ε)O(1/\varepsilon). In this paper, we show that much less space suffices -- we give an algorithm that uses space O(log4ε1)O(\log^4 \varepsilon^{-1}) in the streaming setting while also using an asymptotically optimal number of samples. This is in contrast with the standard total variation distance on discrete distributions for which such space reduction is known to be impossible. Finally, we state 9 related open problems that we hope will spark interest in this and related problems.

Keywords

Cite

@article{arxiv.2410.22123,
  title  = {Testing Identity of Distributions under Kolmogorov Distance in Polylogarithmic Space},
  author = {Christian Janos Lebeda and Jakub Tětek},
  journal= {arXiv preprint arXiv:2410.22123},
  year   = {2024}
}