Building on the success of large language models (LLMs), recent advancements such as GPT-4o have enabled real-time speech interactions through LLM-based voice assistants, offering a significantly improved user experience compared to traditional text-based interactions. However, the absence of benchmarks designed to evaluate these speech interaction capabilities has hindered progress of LLM-based voice assistants development. Current evaluations focus primarily on automatic speech recognition (ASR) or general knowledge evaluation with clean speeches, neglecting the more intricate, real-world scenarios that involve diverse speaker characteristics, environmental and content factors. To address this, we introduce VoiceBench, the first benchmark designed to provide a multi-faceted evaluation of LLM-based voice assistants. VoiceBench also includes both real and synthetic spoken instructions that incorporate the above three key real-world variations. Extensive experiments reveal the limitations of current LLM-based voice assistant models and offer valuable insights for future research and development in this field.
@article{arxiv.2410.17196,
title = {VoiceBench: Benchmarking LLM-Based Voice Assistants},
author = {Yiming Chen and Xianghu Yue and Chen Zhang and Xiaoxue Gao and Robby T. Tan and Haizhou Li},
journal= {arXiv preprint arXiv:2410.17196},
year = {2024}
}
Comments
Work in progress. Data is available at https://github.com/MatthewCYM/VoiceBench