Retention Induced Biases in a Recommendation System with Heterogeneous Users
Abstract
I examine a conceptual model of a recommendation system (RS) with user inflow and churn dynamics. When inflow and churn balance out, the user distribution reaches a steady state. Changing the recommendation algorithm alters the steady state and creates a transition period. During this period, the RS behaves differently from its new steady state. In particular, A/B experiment metrics obtained in transition periods are biased indicators of the RS's long-term performance. Scholars and practitioners, however, often conduct A/B tests shortly after introducing new algorithms to validate their effectiveness. This A/B experiment paradigm, widely regarded as the gold standard for assessing RS improvements, may consequently yield false conclusions. I also briefly touch on the data bias caused by the user retention dynamics.
Keywords
Cite
@article{arxiv.2402.13959,
title = {Retention Induced Biases in a Recommendation System with Heterogeneous Users},
author = {Shichao Ma},
journal= {arXiv preprint arXiv:2402.13959},
year = {2024}
}
Comments
This preprint has not undergone peer review (when applicable) or any post-submission improvements or corrections. The Version of Record of this contribution is published in advances in Bias and Fairness in Information Retrieval. BIAS 2024. Communications in Computer and Information Science, vol 2227. Springer, and is available online at https://doi.org/10.1007/978-3-031-71975-2_2