Clinicians Are Using ChatGPT. How? Why? Now What? It's a Warning.

By Michele Alexander · Founder, MDA Solutions LLC · HIPAA AI Compliance & C-Suite Advisory

Research sourced from: ECRI 2026 Hazards Report, Stanford/Harvard LLM Analysis, JUCM, PMC/NCBI, Read Laboratories, Medscape · Perplexity AI research synthesis, 2026

Someone overheard it at a major hospital, at a clinical station, while visiting for an emergency. You know it. Clinicians are trying to figure something out. They reached for a phone and asked ChatGPT. They got the answer they needed, looked at each other, and said: This is it. Can we use this?

That moment was not a workaround. It was not a policy violation; someone snuck in on personal time. It was clinical staff, at a station, in the middle of their work, reaching for the tool that answered them fastest. And it worked.

Here is the irony: that hospital almost certainly had AI. Probably an excellent AI tool, embedded in their EHR, running quietly in the background, flagging drug interactions, calculating readmission risk, imaging anomalies, and closing care gaps in quality dashboards. Healthcare has been AI-adjacent for years. Most clinicians use these tools every day without ever calling them AI. They show up as alerts. Warnings. Suggestions.

But those tools live inside the EHR. Inside the devices and systems built for documentation. When a question arises outside that workflow, between tasks, at the edges of what the approved system covers, here is often nothing there. And into that gap walks a consumer AI that speaks plain English and answers immediately.

"Generative AI is different. It speaks. It drafts. It answers in the kind of plain language that works at a clinical station, mid-workflow, when the approved system is three clicks too slow — a second opinion that doesn't require a consult or a calendar."

That visibility is the feature. It is also exactly where the governance gaps begin.

The Training Gap No One Talks About

Clinicians did not go to school for informatics. They did not learn business process frameworks or technology governance. They went to school to understand patients, pathology, and care. When a powerful new tool is available mid-workflow — right there, on a phone, at the station, they evaluate it the way they evaluate most things: Does this help me solve this problem right now?

Is that reckless? No. It is human.

But generative AI carries risks that are invisible to someone who was never trained to look for them. Did protected health information appear anywhere in that query? Is the AI-assisted note that was manually entered into the medical record traceable to a specific encounter?

Was it an image, a clinical photo, a chart view, a screen, captured on a personal device and submitted to a third-party model? A resident rotating across three hospitals. A physician managing patient loads at multiple clinics. A case manager working remotely. None of these scenarios feels dangerous in the moment. All of them can have real downstream consequences.

This Is No Longer Hypothetical

What I observed firsthand is now showing up in published research, regulatory warnings, and active litigation. This is no longer a future. It is a present one — documented, reported, and escalating.

Clinicians pasting PHI into public AI tools — HIPAA violation Policy guidance and published reports describe clinicians entering identifiable patient details into consumer tools like ChatGPT or browser-based AI extensions that have no Business Associate Agreement in place. Under HIPAA, this constitutes a potential breach and can trigger Office for Civil Rights investigations and significant fines. The tool does not need to cause harm for the violation to be real.

Lawsuits over AI documentation tools — active litigation Patients have filed lawsuits against health systems alleging that AI-powered documentation and scribing tools were deployed without adequate patient notice and without appropriate safeguards — raising concerns about privacy, informed consent, and clinical accuracy. These cases are in active litigation. The legal exposure is real and growing. (Source: Medscape)

ECRI 2026: AI chatbot misuse ranked a top clinical safety hazard ECRI's 2026 Health Technology Hazards report ranked misuse of AI chatbots by clinicians and staff among its top concerns. Documented cases include chatbots providing incorrect procedural guidance, unsafe device-use recommendations, and misleading diagnostic suggestions — content that caused or risked serious patient harm when acted upon without clinical verification.

AI failing to recognize emergencies in urgent care In urgent care settings, AI intake systems have failed to flag high-acuity symptoms — chest pain, stroke indicators — and continued routing patients through administrative workflows instead of escalating to clinical triage. These failures have resulted in delayed emergency care and legal settlements for failure to triage appropriately. (Source: Read Laboratories)

What about the Harvard study that says AI outperforms doctors? A Harvard-led study published in Science (April 2026) found that OpenAI's o1 model matched or exceeded physician performance on triage, diagnosis, and case management tasks across 76 emergency room cases. Impressive. And worth reading carefully. The study's own senior author, Arjun Manrai, was clear: "This does not mean AI replaces doctors." The research was text-based only. Physicians work with imaging, physiological signals, EKGs, patient affect, and context that no text model has seen. The researchers called for rigorous clinical trials before deployment — not a green light for unsupervised use. Better tools are coming. Governance still matters while they get here.

LLM accuracy in clinical contexts: ~22% harmful recommendations A Stanford and Harvard analysis of medical large language models found severely harmful clinical recommendations in approximately 22 percent of evaluated cases. Clinicians who copy AI-generated advice directly into care plans without independent verification are exposing both patients and themselves to significant and measurable risk. (Source: Burns & Wilcox / Stanford-Harvard)

THE SHADOW AI PROBLEM Policy experts now warn that blanket AI bans in hospital and urgent care settings do not eliminate the behavior — they drive it underground. Clinicians continue using personal accounts, unapproved extensions, and consumer tools at the point of care, but now without any organizational visibility into what is being submitted, what is being returned, or what is entering the medical record. A governance framework built on prohibition alone does not reduce risk. It relocates it — and removes your ability to see it. (Source: JUCM)

Here Is What Governance Actually Means

You've heard this before. AI governance. Policy frameworks. Compliance reviews. It starts to sound like more meetings and more paperwork.

Your staff is already there. They have smartphones. Smartphones are already AI platforms. The question is not whether AI is in your building. It is whether you know what is happening when it is.

Governance is a framework your people can actually use. What is approved? What requires review? Where to go when they are not sure. It is training for judgment, not just compliance. And it is a feedback loop so the people closest to care can tell leadership what is working and what is creating risk — without fear of reprisal. They are your true subject matter experts.

Constant review and iteration are not bureaucratic exercises. In a high-stakes clinical environment, they are how you keep the tools that help your people from becoming the tools that expose them.

The best organizations are not waiting. They are building now, not because they have deployed a new AI platform, but because their employees already have.

What happened at that station was a signal. The organizations that hear it as a warning and act will be ahead. Those who treat it as an embarrassment and move on will instead be managing the consequences.

Sources & References

Source	Finding	Link
ECRI 2026 Hazards Report	AI chatbot misuse ranked as the top clinical safety hazard	ecri.org
Stanford / Harvard via Burns & Wilcox	~22% of LLM medical recommendations found severely harmful	burnsandwilcox.com
JUCM	Case for a generative AI acceptable use policy; shadow AI risk	jucm.com
Read Laboratories	AI triage failures and delayed emergency care in urgent care	readlaboratories.com
PMC / NCBI	Misconfigured AI decision support and legal liability in prescribing	pmc.ncbi.nlm.gov
Medscape	Patient lawsuits filed over AI documentation tools	medscape
Harvard Magazine / Science	AI o1 model matches physician performance on ER tasks; researchers urge clinical trials	harvardmagazine.com
Research synthesis	Compiled and synthesized via Perplexity AI, 2026	perplexity.ai

#HEALTHCARE AI #GOVERNANCE #HIPAA COMPLIANCE