Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix
Abstract
Surrogate index approaches have recently become a popular method of estimating longer-term impact from shorter-term outcomes. In this paper, we leverage 1098 test arms from 200 A/B tests at Netflix to empirically investigate to what degree would decisions made using a surrogate index utilizing 14 days of data would align with those made using direct measurement of day 63 treatment effects. Focusing specifically on linear "auto-surrogate" models that utilize the shorter-term observations of the long-term outcome of interest, we find that the statistical inferences that we would draw from using the surrogate index are ~95% consistent with those from directly measuring the long-term treatment effect. Moreover, when we restrict ourselves to the set of tests that would be "launched" (i.e. positive and statistically significant) based on the 63-day directly measured treatment effects, we find that relying instead on the surrogate index achieves 79% and 65% recall.
Cite
@article{arxiv.2311.11922,
title = {Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix},
author = {Vickie Zhang and Michael Zhao and and Maria Dimakopoulou and Anh Le and Nathan Kallus},
journal= {arXiv preprint arXiv:2311.11922},
year = {2024}
}