We show that large pre-trained language models are inherently highly capable of identifying label errors in natural language datasets: simply examining out-of-sample data points in descending order of fine-tuned task loss significantly outperforms more complex error-detection mechanisms proposed in previous work. To this end, we contribute a novel method for introducing realistic, human-originated label noise into existing crowdsourced datasets such as SNLI and TweetNLP. We show that this noise has similar properties to real, hand-verified label errors, and is harder to detect than existing synthetic noise, creating challenges for model robustness. We argue that human-originated noise is a better standard for evaluation than synthetic noise. Finally, we use crowdsourced verification to evaluate the detection of real errors on IMDB, Amazon Reviews, and Recon, and confirm that pre-trained models perform at a 9-36% higher absolute Area Under the Precision-Recall Curve than existing models.
@article{arxiv.2205.12702,
title = {Detecting Label Errors by using Pre-Trained Language Models},
author = {Derek Chong and Jenny Hong and Christopher D. Manning},
journal= {arXiv preprint arXiv:2205.12702},
year = {2022}
}
Comments
18 pages, 10 figures. Accepted to EMNLP 2022; typesetting of this version slightly differs from conference version