English

Evaluation of AI Chatbots for Patient-Specific EHR Questions

Computation and Language 2023-06-06 v1 Artificial Intelligence Information Retrieval
Authors: Alaleh Hamidi , Kirk Roberts
View on arXiv ↗ PDF ↗

Abstract

This paper investigates the use of artificial intelligence chatbots for patient-specific question answering (QA) from clinical notes using several large language model (LLM) based systems: ChatGPT (versions 3.5 and 4), Google Bard, and Claude. We evaluate the accuracy, relevance, comprehensiveness, and coherence of the answers generated by each model using a 5-point Likert scale on a set of patient-specific questions.

Keywords

ai chatbots patient-specific question answering clinical notes large language models evaluation electronic health records

Cite

@article{arxiv.2306.02549,
  title  = {Evaluation of AI Chatbots for Patient-Specific EHR Questions},
  author = {Alaleh Hamidi and Kirk Roberts},
  journal= {arXiv preprint arXiv:2306.02549},
  year   = {2023}
}

Related papers

View all related →
Artificial Intelligence · Computer Science
How Reliable AI Chatbots are for Disease Prediction from Patient Complaints?
Ayesha Siddika Nipu, K M Sajjadul Islam, Praveen Madiraju
2024-05-24
Computation and Language · Computer Science
Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions
Man Luo, Christopher J. Warren, Lu Cheng, Haidar M. Abdul-Muhsin +1
2024-05-28
Other Quantitative Biology · Quantitative Biology
Artificial Empathy: AI based Mental Health
Aditya Naik, Jovi Thomas, Teja Sree Mandava, Himavanth Reddy Vemula
2025-11-03
Computation and Language · Computer Science
Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study
Zhe He, Balu Bhasuran, Qiao Jin, Shubo Tian +5
2024-04-23
Computation and Language · Computer Science
A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalization
Sarvesh Soni, Dina Demner-Fushman
2026-03-31
Computation and Language · Computer Science
ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan +2
2023-06-27
Computers and Society · Computer Science
A Mixed-Methods Evaluation of LLM-Based Chatbots for Menopause
Roshini Deva, Manvi S, Jasmine Zhou, Elizabeth Britton Chahine +4
2025-02-07
Computation and Language · Computer Science
Can LLMs be Good Financial Advisors?: An Initial Study in Personal Decision Making for Optimized Outcomes
Kausik Lakkaraju, Sai Krishna Revanth Vuruma, Vishal Pallagani, Bharath Muppasani +1
2023-07-17
Computation and Language · Computer Science
The impact of responding to patient messages with large language model assistance
Shan Chen, Marco Guevara, Shalini Moningi, Frank Hoebers +11
2023-12-01
Computation and Language · Computer Science
Performance Assessment of ChatGPT vs Bard in Detecting Alzheimer's Dementia
Balamurali B T, Jer-Ming Chen
2024-02-06
Artificial Intelligence · Computer Science
Fine-tuning Large Language Model (LLM) Artificial Intelligence Chatbots in Ophthalmology and LLM-based evaluation using GPT-4
Ting Fang Tan, Kabilan Elangovan, Liyuan Jin, Yao Jie +8
2024-02-16
Information Retrieval · Computer Science
Dynamic Q&A of Clinical Documents with Large Language Models
Ran Elgedawy, Ioana Danciu, Maria Mahbub, Sudarshan Srinivasan
2024-07-03
Computation and Language · Computer Science
Can ChatGPT and Bard Generate Aligned Assessment Items? A Reliability Analysis against Human Performance
Abdolvahab Khademi
2023-05-11
Computation and Language · Computer Science
Large language models provide unsafe answers to patient-posed medical questions
Rachel L. Draelos, Samina Afreen, Barbara Blasko, Tiffany L. Brazile +13
2025-08-06
Computation and Language · Computer Science
Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AI
Mahyar Abbasian, Elahe Khatibi, Iman Azimi, David Oniani +10
2024-03-01
Human-Computer Interaction · Computer Science
Putting ChatGPT's Medical Advice to the (Turing) Test
Oded Nov, Nina Singh, Devin Mann
2023-01-25
Artificial Intelligence · Computer Science
A Scalable Approach to Benchmarking the In-Conversation Differential Diagnostic Accuracy of a Health AI
Deep Bhatt, Surya Ayyagari, Anuruddh Mishra
2024-12-18
Quantitative Methods · Quantitative Biology
An LLM-Powered Clinical Calculator Chatbot Backed by Verifiable Clinical Calculators and their Metadata
Niranjan Kumar, Farid Seifi, Marisa Conte, Allen Flynn
2025-03-25
Artificial Intelligence · Computer Science
Towards Explainable Conversational AI for Early Diagnosis with Large Language Models
Maliha Tabassum, M Shamim Kaiser
2025-12-22
Computation and Language · Computer Science
Can ChatGPT be Your Personal Medical Assistant?
Md. Rafiul Biswas, Ashhadul Islam, Zubair Shah, Wajdi Zaghouani +1
2023-12-20
Human-Computer Interaction · Computer Science
The Typing Cure: Experiences with Large Language Model Chatbots for Mental Health Support
Inhwa Song, Sachin R. Pendse, Neha Kumar, Munmun De Choudhury
2025-05-12
Computation and Language · Computer Science
ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?
Edisa Lozić, Benjamin Štular
2023-10-17
Computation and Language · Computer Science
ChatCounselor: A Large Language Models for Mental Health Support
June M. Liu, Donghao Li, He Cao, Tianhe Ren +2
2023-09-28
Human-Computer Interaction · Computer Science
Toward Human Centered Interactive Clinical Question Answering System
Dina Albassam
2025-05-27
Computation and Language · Computer Science
Comparative Analysis of Large Language Models in Healthcare
Subin Santhosh, Farwa Abbas, Hussain Ahmad, Claudia Szabo
2026-04-14