Recent advances in large language models (LLMs) have led to the development of various evaluation benchmarks. These benchmarks typically rely on a single instruction template for evaluating all LLMs on a specific task. In this paper, we comprehensively analyze the brittleness of results obtained via single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. To improve robustness of the analysis, we propose to evaluate LLMs with a set of diverse prompts instead. We discuss tailored evaluation metrics for specific use cases (e.g., LLM developers vs. developers interested in a specific downstream task), ensuring a more reliable and meaningful assessment of LLM capabilities. We then implement these criteria and conduct evaluations of multiple models, providing insights into the true strengths and limitations of current LLMs.
@article{arxiv.2401.00595,
title = {State of What Art? A Call for Multi-Prompt LLM Evaluation},
author = {Moran Mizrahi and Guy Kaplan and Dan Malkin and Rotem Dror and Dafna Shahaf and Gabriel Stanovsky},
journal= {arXiv preprint arXiv:2401.00595},
year = {2024}
}
Comments
Accepted at TACL; pre-MIT Press publication version