We propose a domain-adapted reward model that works alongside an Offline A/B testing system for evaluating ranking models. This approach effectively measures reward for ranking model changes in large-scale Ads recommender systems, where model-free methods like IPS are not feasible. Our experiments demonstrate that the proposed technique outperforms both the vanilla IPS method and approaches using non-generalized reward models.
@article{arxiv.2409.19824,
title = {Counterfactual Evaluation of Ads Ranking Models through Domain Adaptation},
author = {Mohamed A. Radwan and Himaghna Bhattacharjee and Quinn Lanners and Jiasheng Zhang and Serkan Karakulak and Houssam Nassif and Murat Ali Bayir},
journal= {arXiv preprint arXiv:2409.19824},
year = {2024}
}
Comments
Accepted at the CONSEQUENCES'24 workshop, co-located with ACM RecSys'24