English

Gaussian credible intervals in Bayesian nonparametric estimation of the unseen

Methodology 2025-01-28 v1 Machine Learning Machine Learning Other Statistics

Abstract

The unseen-species problem assumes n1n\geq1 samples from a population of individuals belonging to different species, possibly infinite, and calls for estimating the number Kn,mK_{n,m} of hitherto unseen species that would be observed if m1m\geq1 new samples were collected from the same population. This is a long-standing problem in statistics, which has gained renewed relevance in biological and physical sciences, particularly in settings with large values of nn and mm. In this paper, we adopt a Bayesian nonparametric approach to the unseen-species problem under the Pitman-Yor prior, and propose a novel methodology to derive large mm asymptotic credible intervals for Kn,mK_{n,m}, for any n1n\geq1. By leveraging a Gaussian central limit theorem for the posterior distribution of Kn,mK_{n,m}, our method improves upon competitors in two key aspects: firstly, it enables the full parameterization of the Pitman-Yor prior, including the Dirichlet prior; secondly, it avoids the need of Monte Carlo sampling, enhancing computational efficiency. We validate the proposed method on synthetic and real data, demonstrating that it improves the empirical performance of competitors by significantly narrowing the gap between asymptotic and exact credible intervals for any m1m\geq1.

Keywords

Cite

@article{arxiv.2501.16008,
  title  = {Gaussian credible intervals in Bayesian nonparametric estimation of the unseen},
  author = {Claudia Contardi and Emanuele Dolera and Stefano Favaro},
  journal= {arXiv preprint arXiv:2501.16008},
  year   = {2025}
}

Comments

63 pages, 5 figures