English

Estimating the size of a set using cascading exclusion

Statistics Theory 2026-04-28 v4 Information Theory math.IT Probability Machine Learning Statistics Theory

Abstract

Let SS be a finite set, and X1,,XnX_1,\ldots,X_n an i.i.d. uniform sample from SS. To estimate the size S|S|, without further structure, one can wait for repeats and use the birthday problem. This requires a sample size of the order S12|S|^\frac{1}{2}. On the other hand, if S={1,2,,S}S=\{1,2,\ldots,|S|\}, the maximum of the sample blown up by n/(n1)n/(n-1) gives an efficient estimator based on any growing sample size. This paper gives refinements that interpolate between these extremes. A general non-asymptotic theory is developed. This includes estimating the volume of a compact convex set, the unseen species problem, and a host of testing problems that follow from the question `Is this new observation a typical pick from a large prespecified population?' We also treat regression style predictors. A general theorem gives non-parametric finite nn error bounds in all cases.

Keywords

Cite

@article{arxiv.2508.05901,
  title  = {Estimating the size of a set using cascading exclusion},
  author = {Sourav Chatterjee and Persi Diaconis and Susan Holmes},
  journal= {arXiv preprint arXiv:2508.05901},
  year   = {2026}
}

Comments

52 pages, 10 figures. Minor changes in this revision. To appear in Statistical Science

R2 v1 2026-07-01T04:40:05.976Z