眼科学LLMs评估基准(BELO)
计算与语言
2025-07-22 v1 人工智能
摘要
当前评估大语言模型(LLMs)在眼科领域的基准受限于范围，且过度侧重准确性。我们引入BELO(眼科BEnchmarking LLMs)，经由13名眼科学家多轮专家审核后形成的标准化、综合性评估基准。BELO评估眼科相关临床准确性与推理质量。采用关键词匹配及微调的PubMedBERT模型，从BCSC、MedMCQA、MedQA、BioASQ及PubMedQA等多样化医学数据集中筛选眼科专用多选题(MCQs)，随后经多轮专家审核剔除重复及次优题目。十名眼科学家细化每道题目正确答案的解释，并由三名资深眼科学家最终裁决。为展示BELO实用性，我们评估了六种LLMs(OpenAI o1、o3-mini、GPT-4o、DeepSeek-R1、Llama-3-8B及Gemini 1.5 Pro)，使用准确率、宏F1及五项文本生成指标(ROUGE-L、BERTScore、BARTScore、METEOR及AlignScore)。此外，纯人工评估中，两名眼科专家对50个随机抽取输出进行准确性、完整性及全面性的质性审查。BELO包含900道高质量、经专家审核的题目，汇自五个来源：BCSC(260题)、BioASQ(10题)、MedMCQA(572题)、MedQA(40题)及PubMedQA(18题)。已建立公开排行榜以促进透明评估与报告。重要的是，BELO数据集将作为holds-out、仅限评估的基准，确保未来模型公平可重现比较。
引用
@article{arxiv.2507.15717,
title = {BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning},
author = {Sahana Srinivasan and Xuguang Ai and Thaddaeus Wai Soon Lo and Aidan Gilson and Minjie Zou and Ke Zou and Hyunjae Kim and Mingjia Yang and Krithi Pushpanathan and Samantha Yew and Wan Ting Loke and Jocelyn Goh and Yibing Chen and Yiming Kong and Emily Yuelei Fu and Michelle Ongyong Hui and Kristen Nwanyanwu and Amisha Dave and Kelvin Zhenghao Li and Chen-Hsin Sun and Mark Chia and Gabriel Dawei Yang and Wendy Meihua Wong and David Ziyou Chen and Dianbo Liu and Maxwell Singer and Fares Antaki and Lucian V Del Priore and Jost Jonas and Ron Adelman and Qingyu Chen and Yih-Chung Tham},
journal= {arXiv preprint arXiv:2507.15717},
year = {2025}
}