Seller-Side Experiments under Interference Induced by Feedback Loops in Two-Sided Platforms
Abstract
Two-sided platforms are central to modern commerce and content sharing and often utilize A/B testing for developing new features. While user-side experiments are common, seller-side experiments become crucial for specific interventions and metrics. This paper investigates the effects of interference caused by feedback loops on seller-side experiments in two-sided platforms, with a particular focus on the counterfactual interleaving design, proposed in \citet{ha2020counterfactual,nandy2021b}. These feedback loops, often generated by pacing algorithms, cause outcomes from earlier sessions to influence subsequent ones. This paper contributes by creating a mathematical framework to analyze this interference, theoretically estimating its impact, and conducting empirical evaluations of the counterfactual interleaving design in real-world scenarios. Our research shows that feedback loops can result in misleading conclusions about the treatment effects.
Keywords
Cite
@article{arxiv.2401.15811,
title = {Seller-Side Experiments under Interference Induced by Feedback Loops in Two-Sided Platforms},
author = {Zhihua Zhu and Zheng Cai and Liang Zheng and Nian Si},
journal= {arXiv preprint arXiv:2401.15811},
year = {2024}
}