Divide-and-conquer methods for big data analysis
Abstract
In the context of big data analysis, the divide-and-conquer methodology refers to a multiple-step process: first splitting a data set into several smaller ones; then analyzing each set separately; finally combining results from each analysis together. This approach is effective in handling large data sets that are unsuitable to be analyzed entirely by a single computer due to limits either from memory storage or computational time. The combined results will provide a statistical inference which is similar to the one from analyzing the entire data set. This article reviews some recently developments of divide-and-conquer methods in a variety of settings, including combining based on parametric, semiparametric and nonparametric models, online sequential updating methods, among others. Theoretical development on the efficiency of the divide-and-conquer methods is also discussed.
Cite
@article{arxiv.2102.10771,
title = {Divide-and-conquer methods for big data analysis},
author = {Xueying Chen and Jerry Q. Cheng and Min-ge Xie},
journal= {arXiv preprint arXiv:2102.10771},
year = {2021}
}