Nonparametric Heterogeneity Testing For Massive Data
Abstract
A massive dataset often consists of a growing number of (potentially) heterogeneous sub-populations. This paper is concerned about testing various forms of heterogeneity arising from massive data. In a general nonparametric framework, a set of testing procedures are designed to accommodate a growing number of sub-populations, denoted as , with computational feasibility. In theory, their null limit distributions are derived as being nearly Chi-square with diverging degrees of freedom as long as does not grow too fast. Interestingly, we find that a lower bound on needs to be set for obtaining a sufficiently powerful testing result, so-called "blessing of aggregation." As a by-produc, a type of homogeneity testing is also proposed with a test statistic being aggregated over all sub-populations. Numerical results are presented to support our theory.
Cite
@article{arxiv.1601.06212,
title = {Nonparametric Heterogeneity Testing For Massive Data},
author = {Junwei Lu and Guang Cheng and Han Liu},
journal= {arXiv preprint arXiv:1601.06212},
year = {2016}
}