English

Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education

Computation and Language 2025-12-01 v4 Artificial Intelligence

Abstract

Background: Simulated patient systems are important in medical education and research, providing safe, integrative training environments and supporting clinical decision making. Advances in artificial intelligence (AI), especially large language models (LLMs), can enhance simulated patients by replicating medical conditions and doctor patient interactions with high fidelity and at low cost, but effectiveness and trustworthiness remain open challenges. Methods: We developed AIPatient, a simulated patient system powered by LLM based AI agents. The system uses a retrieval augmented generation (RAG) framework with six task specific agents for complex reasoning. To improve realism, it is linked to the AIPatient knowledge graph built from de identified real patient data in the MIMIC III intensive care database. Results: We evaluated electronic health record (EHR) based medical question answering (QA), readability, robustness, stability, and user experience. AIPatient reached 94.15 percent QA accuracy when all six agents were enabled, outperforming versions with partial or no agent integration. The knowledge base achieved an F1 score of 0.89. Readability scores showed a median Flesch Reading Ease of 68.77 and a median Flesch Kincaid Grade of 6.4, indicating accessibility for most medical trainees and clinicians. Robustness and stability were supported by non significant variance in repeated trials (analysis of variance F value 0.61, p greater than 0.1; F value 0.78, p greater than 0.1). A user study with medical students showed that AIPatient provides high fidelity, usability, and educational value, comparable to or better than human simulated patients for history taking. Conclusions: LLM based simulated patient systems can deliver accurate, readable, and reliable medical encounters and show strong potential to transform medical education.

Keywords

Cite

@article{arxiv.2409.18924,
  title  = {Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education},
  author = {Huizi Yu and Jiayan Zhou and Lingyao Li and Shan Chen and Jack Gallifant and Anye Shi and Xiang Li and Jingxian He and Wenyue Hua and Mingyu Jin and Guang Chen and Yang Zhou and Zhao Li and Trisha Gupte and Ming-Li Chen and Zahra Azizi and Qi Dou and Bryan P. Yan and Yongfeng Zhang and Yanqiu Xing and Themistocles L. Danielle S. Bitterman and Themistocles L. Assimes and Xin Ma and Lin Lu and Lizhou Fan},
  journal= {arXiv preprint arXiv:2409.18924},
  year   = {2025}
}

Comments

19 pages, 6 figures, 4 tables