English

PARADISE: A Framework for Evaluating Spoken Dialogue Agents

cmp-lg 2008-02-03 v1 Computation and Language
Authors: Marilyn A. Walker , Diane J. Litman , Candace A. Kamm , Alicia Abella
View on arXiv ↗ PDF ↗

Abstract

This paper presents PARADISE (PARAdigm for DIalogue System Evaluation), a general framework for evaluating spoken dialogue agents. The framework decouples task requirements from an agent's dialogue behaviors, supports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to performance, and makes it possible to compare agents performing different tasks by normalizing for task complexity.

Keywords

dialogue systems

Cite

@article{arxiv.cmp-lg/9704004,
  title  = {PARADISE: A Framework for Evaluating Spoken Dialogue Agents},
  author = {Marilyn A. Walker and Diane J. Litman and Candace A. Kamm and Alicia Abella},
  journal= {arXiv preprint arXiv:cmp-lg/9704004},
  year   = {2008}
}

Comments

10 pages, uses aclap, psfig, lingmacros, times

Related papers

View all related →
Computation and Language · Computer Science
Modeling Performance in Open-Domain Dialogue with PARADISE
Marilyn Walker, Colin Harmon, James Graupera, Davan Harrison +1
2021-10-22
cmp-lg · Computer Science
Evaluating Competing Agent Strategies for a Voice Email Agent
Marilyn Walker, Donald Hindle, Jeanne Fromer, Giuseppe Di Fabbrizio +1
2008-02-03
Computation and Language · Computer Science
PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset
Arda Uzunoglu, Abdalfatah Rashid Safa, Gözde Gül Şahin
2024-06-07
Computation and Language · Computer Science
Towards a Progression-Aware Autonomous Dialogue Agent
Abraham Sanders, Tomek Strzalkowski, Mei Si, Albert Chang +3
2022-05-12
Computation and Language · Computer Science
Survey on Evaluation Methods for Dialogue Systems
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen +3
2020-06-29
Computation and Language · Computer Science
Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols
Sarah E. Finch, Jinho D. Choi
2020-06-12
Computation and Language · Computer Science
RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems
Baolin Peng, Chunyuan Li, Zhu Zhang, Chenguang Zhu +2
2021-01-01
Artificial Intelligence · Computer Science
An Application of Reinforcement Learning to Dialogue Strategy Selection in a Spoken Dialogue System for Email
M. A. Walker
2011-06-02
Computation and Language · Computer Science
MoralDial: A Framework to Train and Evaluate Moral Dialogue Systems via Moral Discussions
Hao Sun, Zhexin Zhang, Fei Mi, Yasheng Wang +5
2023-05-29
Artificial Intelligence · Computer Science
Dialectical Reconciliation via Structured Argumentative Dialogues
Stylianos Loukas Vasileiou, Ashwin Kumar, William Yeoh, Tran Cao Son +1
2024-08-09
Computation and Language · Computer Science
How to Evaluate Your Dialogue Models: A Review of Approaches
Xinmeng Li, Wansen Wu, Long Qin, Quanjun Yin
2021-08-04
Computation and Language · Computer Science
Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems
Shivani Kumar, Sumit Bhatia, Milan Aggarwal, Tanmoy Chakraborty
2024-05-24
Computation and Language · Computer Science
Assessing Dialogue Systems with Distribution Distances
Jiannan Xiang, Yahui Liu, Deng Cai, Huayang Li +2
2021-05-28
Computation and Language · Computer Science
On Evaluating and Comparing Open Domain Dialog Systems
Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo +9
2018-12-31
Computation and Language · Computer Science
Dialogue System: A Brief Review
Suket Arora, Kamaljeet Batra, Sarabjit Singh
2013-06-19
cmp-lg · Computer Science
Dialogos: a Robust System for Human-Machine Spoken Dialogue on the Telephone
Dario Albesano, Paolo Baggia, Morena Danieli, Roberto Gemello +2
2008-02-03
Computation and Language · Computer Science
VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue
Yunshui Li, Binyuan Hui, Zhaochao Yin, Wanwei He +5
2023-09-15
Computation and Language · Computer Science
Frame Interpretation and Validation in a Open Domain Dialogue System
Artur Ventura, Nuno Diegues, David Martins de Matos
2012-07-19
Computation and Language · Computer Science
Manual-Guided Dialogue for Flexible Conversational Agents
Ryuichi Takanobu, Hao Zhou, Yankai Lin, Peng Li +2
2022-08-17
Computation and Language · Computer Science
Evaluate On-the-job Learning Dialogue Systems and a Case Study for Natural Language Understanding
Mathilde Veron, Sophie Rosset, Olivier Galibert, Guillaume Bernard
2021-03-01
Computation and Language · Computer Science
BEYOND DIALOGUE: A Profile-Dialogue Alignment Framework Towards General Role-Playing Language Model
Yeyong Yu, Runsheng Yu, Haojie Wei, Zhanqiu Zhang +1
2025-06-10
cmp-lg · Computer Science
Metrics for Evaluating Dialogue Strategies in a Spoken Language System
Morena Danieli, Elisabetta Gerbino
2008-02-03
Computation and Language · Computer Science
On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona +4
2016-06-03
Artificial Intelligence · Computer Science
Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvement
Jan Deriu, Mark Cieliebak
2020-06-26
Computation and Language · Computer Science
DialogueAgents: A Hybrid Agent-Based Speech Synthesis Framework for Multi-Party Dialogue
Xiang Li, Duyi Pan, Hongru Xiao, Jiale Han +4
2025-04-22