BiGGen 基准：用于语言模型之间精细评估的原则性基准
计算与语言
2025-03-26 v2
摘要
随着语言模型(LM)能够处理越来越多的任务，其评估正变得与开发同样具有挑战性。目前大多数生成基准使用像有用性和无害性等抽象评估标准，这些标准往往缺乏人工评估的灵活性和细粒度。此外，这些基准倾向于过度关注特定能力，如指令遵循，从而导致覆盖偏差。为克服这些限制，我们提出 BiGGen 基准，这是一个原则性的生成基准，旨在全面评估 LM 的九种不同能力，涵盖 77 个多样化任务。BiGGen 基准的关键特征是使用任务特定的评估标准，紧密模拟人类评估的细致判断。我们使用五个评估 LM 对 103 个前沿 LM 进行评估。我们的代码、数据和评估结果均已公开发布于 https://github.com/prometheus-eval/prometheus-eval/tree/main/BiGGen-Bench。
引用
@article{arxiv.2406.05761,
title = {The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models},
author = {Seungone Kim and Juyoung Suk and Ji Yong Cho and Shayne Longpre and Chaeeun Kim and Dongkeun Yoon and Guijin Son and Yejin Cho and Sheikh Shafayat and Jinheon Baek and Sue Hyun Park and Hyeonbin Hwang and Jinkyung Jo and Hyowon Cho and Haebin Shin and Seongyun Lee and Hanseok Oh and Noah Lee and Namgyu Ho and Se June Joo and Miyoung Ko and Yoonjoo Lee and Hyungjoo Chae and Jamin Shin and Joel Jang and Seonghyeon Ye and Bill Yuchen Lin and Sean Welleck and Graham Neubig and Moontae Lee and Kyungjae Lee and Minjoon Seo},
journal= {arXiv preprint arXiv:2406.05761},
year = {2025}
}
备注
NAACL 2025 (Main Conference)