Challenges around collecting and processing quality data have hampered progress in data-driven dialogue models. Previous approaches are moving away from costly, resource-intensive lab settings, where collection is slow but where the data is deemed of high quality. The advent of crowd-sourcing platforms, such as Amazon Mechanical Turk, has provided researchers with an alternative cost-effective and rapid way to collect data. However, the collection of fluid, natural spoken or textual interaction can be challenging, particularly between two crowd-sourced workers. In this study, we compare the performance of dialogue models for the same interaction task but collected in two different settings: in the lab vs. crowd-sourced. We find that fewer lab dialogues are needed to reach similar accuracy, less than half the amount of lab data as crowd-sourced data. We discuss the advantages and disadvantages of each data collection method.
@article{arxiv.2012.03855,
title = {The Lab vs The Crowd: An Investigation into Data Quality for Neural Dialogue Models},
author = {José Lopes and Francisco J. Chiyah Garcia and Helen Hastie},
journal= {arXiv preprint arXiv:2012.03855},
year = {2020}
}
Comments
Accepted at Human in the Loop Dialogue Systems Workshop @NeurIPS 2020