English

UserSimCRS v2: Simulation-Based Evaluation for Conversational Recommender Systems

Information Retrieval 2026-03-18 v3

Abstract

Resources for simulation-based evaluation of conversational recommender systems (CRSs) are scarce. The UserSimCRS toolkit was introduced to address this gap. In this work, we present UserSimCRS v2, a significant upgrade aligning the toolkit with state-of-the-art research. Key extensions include an enhanced agenda-based user simulator, introduction of large language model-based simulators, integration for a wider range of CRSs and datasets, and new LLM-as-a-judge evaluation utilities. We demonstrate these extensions in a case study.

Keywords

Cite

@article{arxiv.2512.04588,
  title  = {UserSimCRS v2: Simulation-Based Evaluation for Conversational Recommender Systems},
  author = {Nolwenn Bernard and Krisztian Balog},
  journal= {arXiv preprint arXiv:2512.04588},
  year   = {2026}
}

Comments

Proceedings of the 48th European Conference on Information Retrieval (ECIR '26), 2026

R2 v1 2026-07-01T08:09:06.767Z