Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifesto
Computation and Language
2025-12-23 v2 Human-Computer Interaction
Information Retrieval
Abstract
During the workshop, we deeply discussed what CONversational Information ACcess (CONIAC) is and its unique features, proposing a world model abstracting it, and defined the Conversational Agents Framework for Evaluation (CAFE) for the evaluation of CONIAC systems, consisting of six major components: 1) goals of the system's stakeholders, 2) user tasks to be studied in the evaluation, 3) aspects of the users carrying out the tasks, 4) evaluation criteria to be considered, 5) evaluation methodology to be applied, and 6) measures for the quantitative criteria chosen.
Cite
@article{arxiv.2506.11112,
title = {Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifesto},
author = {Christine Bauer and Li Chen and Nicola Ferro and Norbert Fuhr and Avishek Anand and Timo Breuer and Guglielmo Faggioli and Ophir Frieder and Hideo Joho and Jussi Karlgren and Johannes Kiesel and Bart P. Knijnenburg and Aldo Lipani and Lien Michiels and Andrea Papenmeier and Maria Soledad Pera and Mark Sanderson and Scott Sanner and Benno Stein and Johanne R. Trippas and Karin Verspoor and Martijn C Willemsen},
journal= {arXiv preprint arXiv:2506.11112},
year = {2025}
}
Comments
10 figures; Dagstuhl Manifestos, 11(1), pp 19-67. DOI: 10.4230/DagMan.11.1.19