Related papers: Trouble shooting for covariance fitting in highly …

We address a frequently asked question on the covariance fitting of the highly correlated data such as our $B_K$ data based on the SU(2) staggered chiral perturbation theory. Basically, the essence of the problem is that we do not have an…

We discuss fitting correlated data - with the example of hadron mass spectroscopy in mind. The main conclusion is that the method of minimising correlated $\chi^2$ is unreliable if the data sample is too small.

In optical and infrared long-baseline interferometry, data often display significant correlated errors because of uncertain multiplicative factors such as the instrumental transfer function or the pixel-to-visibility matrix. In the context…

Regression with $\chi^2$ constructed from the covariance matrix should not be used for some combinations of covariance matrices and fitting functions. Using the technique for unsuitable combinations can amplify systematic errors. This…

Estimating a sparse covariance matrix is a fundamental problem in high-dimensional statistics. However, thresholding methods developed for independent data are generally not directly applicable to high-dimensional time series, where…

The variance--covariance matrix plays a central role in the inferential theories of high-dimensional factor models in finance and economics. Popular regularization methods of directly exploiting sparsity are not directly applicable to many…

We discuss fitting hadronic Green functions versus time $t$ to extract mass values in quenched lattice QCD. These data are themselves strongly correlated in $t$. With only a limited number of data samples, the method of minimising…

For high dimensional data, some of the standard statistical techniques do not work well. So modification or further development of statistical methods are necessary. In this paper, we explore these modifications. We start with the important…

High-dimensional compositional data arise naturally in many applications such as metagenomic data analysis. The observed data lie in a high-dimensional simplex, and conventional statistical methods often fail to produce sensible results due…

This paper studies model selection consistency for high dimensional sparse regression when data exhibits both cross-sectional and serial dependency. Most commonly-used model selection methods fail to consistently recover the true model when…

Data analysis in cosmology requires reliable covariance matrices. Covariance matrices derived from numerical simulations often require a very large number of realizations to be accurate. When a theoretical model for the covariance matrix…

One of the goals in scaling sequential machine learning methods pertains to dealing with high-dimensional data spaces. A key related challenge is that many methods heavily depend on obtaining the inverse covariance matrix of the data. It is…

Cross-correlation techniques provide a promising avenue for calibrating photometric redshifts and determining redshift distributions using spectroscopy which is systematically incomplete (e.g., current deep spectroscopic surveys fail to…

Feature screening is an important method to reduce the dimension and capture informative variables in ultrahigh-dimensional data analysis. Many methods have been developed for feature screening. These methods, however, are challenged by…

Estimating a covariance matrix is central to high-dimensional data analysis. Empirical analyses of high-dimensional biomedical data, including genomics, proteomics, microbiome, and neuroimaging, among others, consistently reveal strong…

Observables in particle physics and specifically in lattice QCD calculations are often extracted from fits. Standard $\chi^2$ tests require a reliable determination of the covariance matrix and its inverse from correlated and…

Calibrating out per-antenna signal chain effects is an essential step in analyzing radio interferometric data. For drift-scanning arrays, robustly calibrating the data is especially challenging due to the lack of the ability to track a…

Missing data occur frequently in a wide range of applications. In this paper, we consider estimation of high-dimensional covariance matrices in the presence of missing observations under a general missing completely at random model in the…

Mean field approach, although a generally reliable tool that captures major short range correlations, often fails in symmetric low dimensional strongly correlated electronic systems like those described by the Hubbard model. In these…

Hierarchical inference in (generalized) regression problems is powerful for finding significant groups or even single covariates, especially in high-dimensional settings where identifiability of the entire regression parameter vector may be…