Evaluating Out-of-Distribution Detectors Through Adversarial Generation of Outliers
Abstract
A reliable evaluation method is essential for building a robust out-of-distribution (OOD) detector. Current robustness evaluation protocols for OOD detectors rely on injecting perturbations to outlier data. However, the perturbations are unlikely to occur naturally or not relevant to the content of data, providing a limited assessment of robustness. In this paper, we propose Evaluation-via-Generation for OOD detectors (EvG), a new protocol for investigating the robustness of OOD detectors under more realistic modes of variation in outliers. EvG utilizes a generative model to synthesize plausible outliers, and employs MCMC sampling to find outliers misclassified as in-distribution with the highest confidence by a detector. We perform a comprehensive benchmark comparison of the performance of state-of-the-art OOD detectors using EvG, uncovering previously overlooked weaknesses.
Keywords
Cite
@article{arxiv.2208.10940,
title = {Evaluating Out-of-Distribution Detectors Through Adversarial Generation of Outliers},
author = {Sangwoong Yoon and Jinwon Choi and Yonghyeon Lee and Yung-Kyun Noh and Frank Chongwoo Park},
journal= {arXiv preprint arXiv:2208.10940},
year = {2022}
}
Comments
Code release can be found at https://github.com/EvG-OOD/evaluation-via-generation