Regularized $M$-estimators of scatter matrix

In this paper, a general class of regularized $M$-estimators of scatter matrix are proposed which are suitable also for low or insufficient sample support (small $n$ and large $p$) problems. The considered class constitutes a natural generalization of $M$-estimators of scatter matrix (Maronna, 1976) and are defined as a solution to a penalized $M$-estimation cost function that depend on a pair $(\alpha,\beta)$ of regularization parameters. We derive general conditions for uniqueness of the solution using concept of geodesic convexity. Since these conditions do not include Tyler's $M$-estimator, necessary and sufficient conditions for uniqueness of the penalized Tyler's cost function are established separately. For the regularized Tyler's $M$-estimator, we also derive a simple, closed form and data dependent solution for choosing the regularization parameter based on shape matrix matching in the mean squared sense. An iterative algorithm that converges to the solution of the regularized $M$-estimating equation is also provided. Finally, some simulations studies illustrate the improved accuracy of the proposed regularized $M$-estimators of scatter compared to their non-regularized counterparts in low sample support problems. An example of radar detection using normalized matched filter (NMF) illustrate that an adaptive NMF detector based on regularized $M$-estimators are able to maintain accurately the preset CFAR level and at at the same time provide similar probability of detection as the (theoretical) NMF detector.