A Cross-Domain Benchmark for Active Learning
Abstract
Active Learning (AL) deals with identifying the most informative samples for labeling to reduce data annotation costs for supervised learning tasks. AL research suffers from the fact that lifts from literature generalize poorly and that only a small number of repetitions of experiments are conducted. To overcome these obstacles, we propose CDALBench, the first active learning benchmark which includes tasks in computer vision, natural language processing and tabular learning. Furthermore, by providing an efficient, greedy oracle, CDALBench can be evaluated with 50 runs for each experiment. We show, that both the cross-domain character and a large amount of repetitions are crucial for sophisticated evaluation of AL research. Concretely, we show that the superiority of specific methods varies over the different domains, making it important to evaluate Active Learning with a cross-domain benchmark. Additionally, we show that having a large amount of runs is crucial. With only conducting three runs as often done in the literature, the superiority of specific methods can strongly vary with the specific runs. This effect is so strong, that, depending on the seed, even a well-established method's performance can be significantly better and significantly worse than random for the same dataset.
Cite
@article{arxiv.2408.00426,
title = {A Cross-Domain Benchmark for Active Learning},
author = {Thorben Werner and Johannes Burchert and Maximilian Stubbemann and Lars Schmidt-Thieme},
journal= {arXiv preprint arXiv:2408.00426},
year = {2024}
}
Comments
Accepted at NeurIPS 24 in the Benchmarks and Datasets Track. Updated version of paper "Toward Comparable Active Learning" (arXiv:2311.18356). "Toward Comparable Active Learning" is deprecated, please use this version. arXiv admin note: text overlap with arXiv:2311.18356; text overlap with arXiv:2301.10625 by other authors