In recommender systems, online A/B testing is a crucial method for evaluating the performance of different models. However, conducting online A/B testing often presents significant challenges, including substantial economic costs, user experience degradation, and considerable time requirements. With the Large Language Models' powerful capacity, LLM-based agent shows great potential to replace traditional online A/B testing. Nonetheless, current agents fail to simulate the perception process and interaction patterns, due to the lack of real environments and visual perception capability. To address these challenges, we introduce a multi-modal user agent for A/B testing (A/B Agent). Specifically, we construct a recommendation sandbox environment for A/B testing, enabling multimodal and multi-page interactions that align with real user behavior on online platforms. The designed agent leverages multimodal information perception, fine-grained user preferences, and integrates profiles, action memory retrieval, and a fatigue system to simulate complex human decision-making. We validated the potential of the agent as an alternative to traditional A/B testing from three perspectives: model, data, and features. Furthermore, we found that the data generated by A/B Agent can effectively enhance the capabilities of recommendation models. Our code is publicly available at https://github.com/Applied-Machine-Learning-Lab/ABAgent.
@article{arxiv.2601.04554,
title = {Exploring Recommender System Evaluation: A Multi-Modal User Agent Framework for A/B Testing},
author = {Wenlin Zhang and Xiangyang Li and Qiyuan Ge and Kuicai Dong and Pengyue Jia and Xiaopeng Li and Zijian Zhang and Maolin Wang and Yichao Wang and Huifeng Guo and Ruiming Tang and Xiangyu Zhao},
journal= {arXiv preprint arXiv:2601.04554},
year = {2026}
}