Dynamic-SUPERB 第二阶段：一个用于衡量口语语言模型能力的协作扩展基准，包含180项任务
计算与语言
2025-06-10 v2 音频与语音处理
摘要
多模态基础模型，如 Gemini 和 ChatGPT，通过无缝集成多种数据形式，彻底改变了人机交互。开发一种能够理解广泛自然语言指令的通用口语语言模型，对于弥合沟通鸿沟和促进更直观的交互至关重要。然而，缺乏全面的评估基准构成了重大挑战。我们提出了 Dynamic-SUPERB Phase-2，这是一个用于全面评估基于指令的通用语音模型的开放且不断演进的基准。在第一代的基础上，第二版纳入了全球研究社区协作贡献的125项新任务，将基准扩展到总共180项任务，使其成为最大规模的语音与音频评估基准。第一代 Dynamic-SUPERB 仅限于分类任务，而 Dynamic-SUPERB Phase-2 通过引入涵盖语音、音乐和环境音频的广泛新颖多样化任务（包括回归和序列生成），拓宽了其评估能力。评估结果表明，没有任何模型在所有任务上表现优异。SALMONN-13B 在英语 ASR 方面表现出色，Qwen2-Audio-7B-Instruct 在情感识别方面显示出高准确率，但当前模型仍需进一步创新才能处理更广泛的任务。我们已在 https://github.com/dynamic-superb/dynamic-superb 开源了所有任务数据和评估流程。
引用
@article{arxiv.2411.05361,
title = {Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks},
author = {Chien-yu Huang and Wei-Chih Chen and Shu-wen Yang and Andy T. Liu and Chen-An Li and Yu-Xiang Lin and Wei-Cheng Tseng and Anuj Diwan and Yi-Jen Shih and Jiatong Shi and William Chen and Chih-Kai Yang and Wenze Ren and Xuanjun Chen and Chi-Yuan Hsiao and Puyuan Peng and Shih-Heng Wang and Chun-Yi Kuan and Ke-Han Lu and Kai-Wei Chang and Fabian Ritter-Gutierrez and Kuan-Po Huang and Siddhant Arora and You-Kuan Lin and Ming To Chuang and Eunjung Yeo and Kalvin Chang and Chung-Ming Chien and Kwanghee Choi and Jun-You Wang and Cheng-Hsiu Hsieh and Yi-Cheng Lin and Chee-En Yu and I-Hsiang Chiu and Heitor R. Guimarães and Jionghao Han and Tzu-Quan Lin and Tzu-Yuan Lin and Homu Chang and Ting-Wu Chang and Chun Wei Chen and Shou-Jen Chen and Yu-Hua Chen and Hsi-Chun Cheng and Kunal Dhawan and Jia-Lin Fang and Shi-Xin Fang and Kuan-Yu Fang Chiang and Chi An Fu and Hsien-Fu Hsiao and Ching Yu Hsu and Shao-Syuan Huang and Lee Chen Wei and Hsi-Che Lin and Hsuan-Hao Lin and Hsuan-Ting Lin and Jian-Ren Lin and Ting-Chun Liu and Li-Chun Lu and Tsung-Min Pai and Ankita Pasad and Shih-Yun Shan Kuan and Suwon Shon and Yuxun Tang and Yun-Shao Tsai and Jui-Chiang Wei and Tzu-Chieh Wei and Chengxi Wu and Dien-Ruei Wu and Chao-Han Huck Yang and Chieh-Chi Yang and Jia Qi Yip and Shao-Xiang Yuan and Vahid Noroozi and Zhehuai Chen and Haibin Wu and Karen Livescu and David Harwath and Shinji Watanabe and Hung-yi Lee},
journal= {arXiv preprint arXiv:2411.05361},
year = {2025}
}
备注
ICLR 2025