LLM 对输出格式偏见！系统评估与缓解 LLM 输出格式偏见

我们提出了首个系统性评估大语言模型（LLM）输出格式偏见的研究。我们的ethods distinguish between two categories of an evaluation metric under format constraints to reliably and accurately assess performance: one measures performance when format constraints are adhered to, while the other evaluates performance regardless of constraint adherence. 我们 then define a metric for measuring the format bias of LLMs and establish effective strategies to reduce it. subsequently, we present our empirical format bias evaluation spanning four commonly used categories -- multiple-choice question-answer, wrapping, list, and mapping -- covering 15 widely-used formats. our evaluation on eight generation tasks uncovers significant format bias across state-of-the-art LLMs. we further discover that improving the format-instruction following capabilities of LLMs across formats potentially reduces format bias. based on our evaluation findings, we study prompting and fine-tuning with synthesized format data techniques to mitigate format bias. our methods successfully reduce the variance in ChatGPT's performance among wrapping formats from 235.33 to 0.71 (%$^2)。