A Random Matrix--Theoretic Approach to Handling Singular Covariance Estimates

In many practical situations we would like to estimate the covariance matrix of a set of variables from an insufficient amount of data. More specifically, if we have a set of $N$ independent, identically distributed measurements of an $M$ dimensional random vector the maximum likelihood estimate is the sample covariance matrix. Here we consider the case where $N<M$ such that this estimate is singular and therefore fundamentally bad. We present a radically new approach to deal with this situation. Let $X$ be the $M\times N$ data matrix, where the columns are the $N$ independent realizations of the random vector with covariance matrix $\Sigma$. Without loss of generality, we can assume that the random variables have zero mean. We would like to estimate $\Sigma$ from $X$. Let $K$ be the classical sample covariance matrix. Fix a parameter $1\leq L\leq N$ and consider an ensemble of $L\times M$ random unitary matrices, $\{\Phi\}$, having Haar probability measure. Pre and post multiply $K$ by $\Phi$, and by the conjugate transpose of $\Phi$ respectively, to produce a non--singular $L\times L$ reduced dimension covariance estimate. A new estimate for $\Sigma$, denoted by $\mathrm{cov}_L(K)$, is obtained by a) projecting the reduced covariance estimate out (to $M\times M$) through pre and post multiplication by the conjugate transpose of $\Phi$, and by $\Phi$ respectively, and b) taking the expectation over the unitary ensemble. Another new estimate (this time for $\Sigma^{-1}$), $\mathrm{invcov}_L(K)$, is obtained by a) inverting the reduced covariance estimate, b) projecting the inverse out (to $M\times M$) through pre and post multiplication by the conjugate transpose of $\Phi$, and by $\Phi$ respectively, and c) taking the expectation over the unitary ensemble. We have a closed analytical expression for $\mathrm{invcov}_L(K)$ and $\mathrm{cov}_L(K)$ in terms of its eigenvalue decomposition.