Related papers: Nearly optimal capture-recapture sampling and empi…

In a typical two-phase design, a random sample is drawn from the target population in phase 1, during which only a subset of variables is collected. In phase 2, a subsample of the phase-1 cohort is selected, and additional variables are…

This article introduces subbagging (subsample aggregating) estimation approaches for big data analysis with memory constraints of computers. Specifically, for the whole dataset with size $N$, $m_N$ subsamples are randomly drawn, and each…

The era of huge data necessitates highly efficient machine learning algorithms. Many common machine learning algorithms, however, rely on computationally intensive subroutines that are prohibitively expensive on large datasets. Oftentimes,…

The objective of this work is to quantify the uncertainty in probability of failure estimates resulting from incomplete knowledge of the probability distributions for the input random variables. We propose a framework that couples the…

This paper studies empirical risk minimization (ERM) problems for large-scale datasets and incorporates the idea of adaptive sample size methods to improve the guaranteed convergence bounds for first-order stochastic and deterministic…

Datasets with sheer volume have been generated from fields including computer vision, medical imageology, and astronomy whose large-scale and high-dimensional properties hamper the implementation of classical statistical models. To tackle…

Sub-sampling is a common and often effective method to deal with the computational challenges of large datasets. However, for most statistical models, there is no well-motivated approach for drawing a non-uniform subsample. We show that the…

In this paper, we propose new sampling approaches for the Shrinkage Inverse-Wishart (SIW) distribution, a generalized family of the Inverse-Wishart distribution originally proposed by Berger et al. (2020, Annals of Statistics). It offers a…

In capture-recapture experiments, individual covariates may be subject to missing, especially when the number of times of being captured is small. When the covariate information is missing at random, the inverse probability weighting method…

With the growing availability of large-scale biomedical data, it is often time-consuming or infeasible to directly perform traditional statistical analysis with relatively limited computing resources at hand. We propose a fast subsampling…

The Mann-Whitney-Wilcoxon rank sum test (MWWRST) is a widely used method for comparing two treatment groups in randomized control trials, particularly when dealing with highly skewed data. However, when applied to observational study data,…

Running machine learning algorithms on large and rapidly growing volumes of data is often computationally expensive, one common trick to reduce the size of a data set, and thus reduce the computational cost of machine learning algorithms,…

The IBOSS approach proposed by Wang et al. (2019) selects the most informative subset of n points. It assumes that the ordinary least squares method is used and requires that the number of variables, p, is not large. However, in many…

The ability to quantify the directional flow of information is vital to understanding natural systems and designing engineered information-processing systems. A widely used measure to quantify this information flow is the transfer entropy.…

As computer resources become increasingly limited, traditional statistical methods face challenges in analyzing massive data, especially in functional data analysis. To address this issue, subsampling offers a viable solution by…

Modern statistical analysis often encounters datasets with large sizes. For these datasets, conventional estimation methods can hardly be used immediately because practitioners often suffer from limited computational resources. In most…

The inverse probability weighting (IPW) method is used to handle attrition in association analyses derived from cohort studies. It consists in weighting the respondents at a given follow-up by their inverse probability to participate.…

We consider the challenges that arise when fitting complex ecological models to 'large' data sets. In particular, we focus on random effect models which are commonly used to describe individual heterogeneity, often present in ecological…

Most epidemiologic cohorts are composed of volunteers who do not represent the general population. To enable population inference from cohorts, we and others have proposed utilizing probability survey samples as external references to…

Bias in datasets can be very detrimental for appropriate statistical estimation. In response to this problem, importance weighting methods have been developed to match any biased distribution to its corresponding target unbiased…