Testing High-dimensional Multinomials with Applications to Text Analysis
Methodology
2023-11-28 v2 Statistics Theory
Machine Learning
Statistics Theory
Abstract
Motivated by applications in text mining and discrete distribution inference, we investigate the testing for equality of probability mass functions of groups of high-dimensional multinomial distributions. A test statistic, which is shown to have an asymptotic standard normal distribution under the null, is proposed. The optimal detection boundary is established, and the proposed test is shown to achieve this optimal detection boundary across the entire parameter space of interest. The proposed method is demonstrated in simulation studies and applied to analyze two real-world datasets to examine variation among consumer reviews of Amazon movies and diversity of statistical paper abstracts.
Keywords
Cite
@article{arxiv.2301.01381,
title = {Testing High-dimensional Multinomials with Applications to Text Analysis},
author = {T. Tony Cai and Zheng Tracy Ke and Paxton Turner},
journal= {arXiv preprint arXiv:2301.01381},
year = {2023}
}