We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings.
@article{arxiv.2601.03627,
title = {Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines},
author = {Jean Seo and Gibaeg Kim and Kihun Shin and Seungseop Lim and Hyunkyung Lee and Wooseok Han and Jongwon Lee and Eunho Yang},
journal= {arXiv preprint arXiv:2601.03627},
year = {2026}
}