A Course Shared Task on Evaluating LLM Output for Clinical Questions

Yufang Hou; Thy Thy Tran; Doan Nam Long Vu; Yiwen Cao; Kai Li; Lukas Rohde; Iryna Gurevych

A Course Shared Task on Evaluating LLM Output for Clinical Questions

Computation and Language 2024-08-02 v1

Authors: Yufang Hou , Thy Thy Tran , Doan Nam Long Vu , Yiwen Cao , Kai Li , Lukas Rohde , Iryna Gurevych

Abstract

This paper presents a shared task that we organized at the Foundations of Language Technology (FoLT) course in 2023/2024 at the Technical University of Darmstadt, which focuses on evaluating the output of Large Language Models (LLMs) in generating harmful answers to health-related clinical questions. We describe the task design considerations and report the feedback we received from the students. We expect the task and the findings reported in this paper to be relevant for instructors teaching natural language processing (NLP) and designing course assignments.

Keywords

large language model large language model evaluation language modeling

Cite

@article{arxiv.2408.00122,
  title  = {A Course Shared Task on Evaluating LLM Output for Clinical Questions},
  author = {Yufang Hou and Thy Thy Tran and Doan Nam Long Vu and Yiwen Cao and Kai Li and Lukas Rohde and Iryna Gurevych},
  journal= {arXiv preprint arXiv:2408.00122},
  year   = {2024}
}

Comments

accepted at the sixth Workshop on Teaching NLP (co-located with ACL 2024)

A Course Shared Task on Evaluating LLM Output for Clinical Questions

Abstract

Keywords

Cite

Comments

Related papers