Exploring the Feasibility and Acceptability of AI-Mediated Serious Illness Conversations in the Emergency Department
Hasibur Rahman, Kenji Numata, Evelyn T Lai, Maria Cheriyan, Adrian Haimovich + 2 more
TLDR
AI-mediated serious illness conversations in the ED show promise for patient values alignment but present critical ethical and emotional risks.
Key contributions
- Developed ED GOAL-AI, a voice-based AI for serious illness conversations in emergency departments.
- Evaluated ED GOAL-AI with 55 patients, finding it feasible and acceptable, comparable to clinicians.
- Observed critical failure modes, including AI hallucinating diagnostic statements and boundary violations.
- Emphasizes careful boundary setting and participatory design for safe AI deployment in healthcare.
Why it matters
Serious illness conversations are vital for patient-aligned care but are often overlooked in busy EDs. This paper explores AI's potential to bridge this gap, showing promise but also critical ethical risks like AI hallucinations. It underscores the necessity of robust design and boundary setting for safe AI integration.
Original Abstract
Serious illness conversations (SICs) align care with patients' values, goals, and preferences, yet they rarely occur in emergency departments (EDs), where time constraints and emotional burden often leave clinicians making high-stakes decisions without documented insight into what matters most to patients. We present a case study of ED GOAL-AI, a voice-based conversational agent for brief, structured values discussions with older adults in the ED, evaluated with 55 patients for feasibility and acceptability. Most participants completed the conversation and reported the interaction as acceptable and feasible, with ratings of feeling heard and understood comparable to clinicians. However, we also observed critical failure modes, including boundary violations such as hallucinated diagnostic statements, highlighting ethical and emotional risks. This work points to early promise for AI-mediated SICs while underscoring the need for careful boundary setting and participatory design before broader deployment.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.