Adaptive Selective Sampling for Online Prediction with Experts
Abstract
We consider online prediction of a binary sequence with expert advice. For this setting, we devise label-efficient forecasting algorithms, which use a selective sampling scheme that enables collecting much fewer labels than standard procedures, while still retaining optimal worst-case regret guarantees. These algorithms are based on exponentially weighted forecasters, suitable for settings with and without a perfect expert. For a scenario where one expert is strictly better than the others in expectation, we show that the label complexity of the label-efficient forecaster scales roughly as the square root of the number of rounds. Finally, we present numerical experiments empirically showing that the normalized regret of the label-efficient forecaster can asymptotically match known minimax rates for pool-based active learning, suggesting it can optimally adapt to benign settings.
Cite
@article{arxiv.2302.08397,
title = {Adaptive Selective Sampling for Online Prediction with Experts},
author = {Rui M. Castro and Fredrik Hellström and Tim van Erven},
journal= {arXiv preprint arXiv:2302.08397},
year = {2023}
}