Real-time outlier detection for large datasets by RT-DetMCD
Abstract
Modern industrial machines can generate gigabytes of data in seconds, frequently pushing the boundaries of available computing power. Together with the time criticality of industrial processing this presents a challenging problem for any data analytics procedure. We focus on the deterministic minimum covariance determinant method (DetMCD), which detects outliers by fitting a robust covariance matrix. We construct a much faster version of DetMCD by replacing its initial estimators by two new methods and incorporating update-based concentration steps. The computation time is reduced further by parallel computing, with a novel robust aggregation method to combine the results from the threads. The speed and accuracy of the proposed real-time DetMCD method (RT-DetMCD) are illustrated by simulation and a real industrial application to food sorting.
Keywords
Cite
@article{arxiv.1910.05615,
title = {Real-time outlier detection for large datasets by RT-DetMCD},
author = {Bart De Ketelaere and Mia Hubert and Jakob Raymaekers and Peter J. Rousseeuw and Iwein Vranckx},
journal= {arXiv preprint arXiv:1910.05615},
year = {2021}
}