Clinical Safety Risks
Multimodal models in clinical settings generate inaccurate, unreliable, or harmful medical information that directly threatens patient safety.
Models generate plausible-sounding but factually incorrect medical content — fabricating pathological findings, producing erroneous diagnoses, or confabulating non-existent references.
The MedVH benchmark revealed that medical fine-tuned LVLMs (LLaVA-Med, Med-Flamingo) perform worse on hallucination tests than general-purpose GPT-4V — a striking "fine-tuning paradox" where domain specialization increases hallucination susceptibility in chest X-ray VQA and report generation.
Gu et al., 2024 — Six-task hallucination evaluation framework; reveals fine-tuning–hallucination tradeoff in domain-specific medical LVLMs.
Zhu et al., ACL 2025 — Taxonomizes medical hallucinations as consistency (misdiagnosis) vs. completeness (missed diagnosis) types; 100+ paper survey.
Xia et al., NeurIPS 2024 — Evaluates across five trust dimensions (truthfulness, fairness, safety, privacy, robustness); 41K QA pairs, 16 modalities.
Survey, 2025 — Clinician survey (n=70): 91.8% encountered medical hallucinations; 84.7% consider them capable of causing patient harm.
Roustan & Bastardot, IJMR 2025 — Practical guide for clinicians on hallucination root causes and their cascade effects on clinical reasoning.
Models systematically amplify rather than flag embedded clinical errors, propagating and elaborating on false information inserted in realistic clinical scenarios.
Communications Medicine (Omar et al., 2025): 300 clinician-designed vignettes each containing one deliberate error. Six leading LLMs repeated or elaborated on the planted error in up to 83% of cases. A mitigation prompt reduced this to ~41% — still clinically unacceptable.
Omar et al., Communications Medicine 2025 — 83% error propagation rate across six leading LLMs using clinician-planted vignettes.
Bedi et al., 2024 — Clinician-led evaluation of GPT-4V across real clinical cases; concludes current multimodal models are not fit for clinical deployment.
npj Digital Medicine, 2025 — 12,999 annotated sentences; 1.47% hallucination rate; 3.45% omission rate — omissions posing greater clinical danger.
Models fail to correctly integrate imaging content with textual context, producing outputs that contradict visual reality — or answering "correctly" without genuinely processing the image.
MediConfusion specifically tests cases clinicians consider diagnostically obvious. Mainstream Med-VLMs produce misleading diagnoses on exactly these unambiguous cases — revealing fundamental cross-modal reasoning failure, not merely knowledge gaps.
2024 — Tests Med-VLMs on clinician-designated "easy" cases; exposes systematic diagnostic failures on unambiguous imaging scenarios.
Yan et al., 2024 — Image-swapping probes show many Med-LVLMs answer without processing the image, relying on text-only biases.
Chen et al., 2023 — Early foundational work on hallucination detection methodology for medical LVLMs; establishes evaluation baselines.
Models have static training cutoffs. They may cite retracted studies, apply superseded guidelines, or recommend outdated dosing standards — with real patient harm potential in rapidly evolving therapeutic areas.
AI warfarin dosing models trained without adequate African-American genetic variant coverage systematically recommend overdoses for this population (CYP2C9 allele underrepresentation) — where "outdated knowledge" directly intersects with racial bias, causing pharmacological harm.
Classifies "Outdated Reference Hallucination" as a major hallucination subtype; models relying on obsolete guidelines or retracted literature.
Multi-institution, 2024 — Jointly published by leading academic institutions; identifies knowledge cutoff as a core safety challenge in rapidly evolving therapeutic areas.
Agarwal et al., 2024 — Evaluates hallucination type distribution in real patient health queries; outdated knowledge identified as substantial error contributor.
Clinicians and patients over-trust AI outputs, suppressing independent judgment. Even when models recommend seeking human consultation, only ~50% of users comply — undermining safety nets.
A systematic review of LLMs in mental health (*PMC*, 2025) across 55 studies identifies the "halo effect" as a primary risk: users systematically overestimate AI accuracy, increasing misdiagnosis and delayed treatment risk. Recommending professional help was followed by only ~half of users.
PMC, 2025 — Quantifies halo effect in 55 mental health AI studies; links to delayed treatment-seeking and therapeutic relationship erosion.
2024 — Examines trust-formation mechanisms when LLMs are embedded in medical social robots; analyzes automation bias induction patterns.
Privacy & Cybersecurity Risks
Medical multimodal models handle highly sensitive patient data, facing end-to-end threats from training-time poisoning to deployment-time adversarial manipulation.
Models memorize and reproduce patient data from training sets. Membership inference attacks — requiring only black-box API access — can determine whether specific patients' data was used for training, enabling sensitive health inference.
Models trained on MIMIC-CXR have been shown to inadvertently reproduce specific patient image-report pairs via API-only querying — violating HIPAA without any internal model access. The attack is feasible with standard black-box query access to deployed APIs.
2024 — Systematically maps medical LLM privacy threats (membership inference, model inversion, attribute inference); surveys defensive techniques.
2025 — First model-stealing attack against medical MLLMs using only natural images; replicates model without any medical data access, raising IP and privacy risks.
JMIR, 2026 — Comprehensive threat analysis synthesizing 41 security studies; addresses privacy exposure in federated learning deployments.
Malicious instructions embedded in DICOM metadata, patient record fields, or report annotations hijack model behavior — redirecting it to leak information or generate dangerous outputs, invisible to clinical staff.
Researchers embedded attack instructions in DICOM metadata fields of a routine chest X-ray. When the oncology VLM processed the image, hidden instructions activated — altering the diagnostic recommendation and transmitting patient names and clinical summaries to an external address. Clinical staff detected nothing anomalous.
2024 — First systematic clinical prompt injection demonstration; shows attacks via imaging metadata can cause information leakage and diagnostic manipulation.
2024 — Classifies prompt injection as the highest-severity vector in medical AI agent systems due to invisibility and cascading clinical consequences.
2025 — Three-strategy poisoning framework for RAG-augmented medical systems; cross-modal conflict injection disrupts retrieval without model access.
Imperceptibly perturbed medical images fool AI into producing catastrophically wrong diagnoses — changes invisible to radiologists can reduce detection accuracy from 95% to near zero.
Finlayson et al. (Science, 2019): Pixel-level perturbations added to chest X-rays caused AI to diagnose pneumonia cases as normal with high confidence. Detection accuracy: 95% → ~0%. Perturbed images were indistinguishable from originals to board-certified radiologists. The paper also modeled realistic adversary motivations (insurance fraud, malpractice cover-up).
Finlayson et al., Science 2019 — Landmark paper demonstrating medical ML adversarial vulnerability; models realistic attacker motivations in healthcare.
Frontiers in Medicine, 2025 — Compares single vs. multimodal medical model robustness; multimodal fusion partially improves robustness but introduces new attack surfaces.
npj Digital Medicine, 2024 — Modifying 1.1% of model weights stably injects false biomedical facts; errors propagate while other task performance is preserved.
Attackers embed "backdoor triggers" during training. The deployed model behaves normally on clean inputs but produces attacker-specified outputs when a specific trigger condition is met — enabling silent, targeted medical discrimination.
JMIR (2026) models a technically grounded attack scenario: an adversary poisons an organ transplant prioritization AI to systematically lower priority scores for a specific demographic group. Current transplant oversight — monitoring aggregate outcomes, not AI forensics — cannot detect this. The bias becomes statistically visible only after years of disparate transplant outcomes.
JMIR, 2026 — Eight technically grounded attack scenarios across healthcare AI architectures; analyzes detectability gaps in current clinical oversight systems.
2024 — Analyzes backdoor propagation in multi-agent medical systems; a compromised sub-agent can silently spread poisoned outputs through entire clinical workflows.
Algorithmic Bias & Health Equity
Medical multimodal models learn and amplify social biases from historical data, producing systematic diagnostic disparities that harm marginalized populations and deepen healthcare inequality.
Training data severely underrepresents dark skin tones; models perform substantially worse for dermatological and pathological imaging tasks on darker-skinned patients, causing missed cancers and diagnostic delays.
Oxford researchers (*Lancet Digital Health*) examined 21 open-access skin cancer AI datasets (106,950 images total). Only 2,436 recorded skin type. Among those: 10 images with brown skin, 1 image with dark/black skin. Experts warned this means widely deployed AI "may cause avoidable surgery, missed treatable cancers, and unnecessary anxiety" for patients with dark skin.
Science Advances, 2024 — Expert-level medical VLMs show consistent underdiagnosis bias for Black patients vs. certified radiologists on chest X-ray datasets.
Seyyed-Kalantari et al., Nature Medicine 2021 — Benchmark across three chest X-ray datasets; highest underdiagnosis for intersectional groups (Hispanic female patients).
npj Digital Medicine, 2025 — Systematic review of bias sources and their propagation across the full medical AI lifecycle; mitigation strategy catalog.
Historical medical data is male-dominated. Models underperform on female-specific symptom presentations and may reproduce historically discriminatory clinical practices in their recommendations.
U.S. Senator Ben Ray Luján cited congressional testimony: medical AI trained predominantly on male data performed substantially worse on female patients. A 2024 UK government-commissioned review (Equity in Medical Devices) found women, minority ethnic groups, and people from deprived communities systematically receive lower-quality care from biased medical AI tools.
Journal of Clinical Epidemiology, 2024 — Gender bias identified as one of the most pervasive bias types across clinical ML domains.
PMC, 2021 — Documents systematic LGBTQ+ data absence in EHR systems; AI is blind to health disparities affecting this population.
Medical AI training data originates predominantly from high-income urban hospitals. Models fail to generalize to low-resource settings, rural environments, or different equipment standards — amplifying global health disparities.
A 2024 mixed-methods study found that over 90% of medical AI fairness research focuses on Western contexts. African nations face a dual disadvantage: inadequate local data representation and colonial-legacy infrastructure disparities — making Western-trained AI systems systematically unreliable in these settings, largely invisible in mainstream AI safety discourse.
2024 — Applies postcolonial theory to medical AI fairness; reveals how colonial power structures translate into current data representation failures.
Frontiers in Public Health, 2025 — Multi-stage analysis of bias sources in low-resource medical settings; setting-specific mitigation strategies.
Removing protected attributes (race, gender) from training data is insufficient — models exploit correlated proxies (zip code, insurance type, historical cost) to reconstruct and amplify historical discrimination patterns.
Obermeyer et al. (Science, 2019): A widely deployed U.S. health management algorithm used "historical healthcare costs" as a proxy for medical need. Since Black patients historically received less care (due to systemic inequality), they were assigned lower health risk scores than equally sick white patients — at scale across millions of patients, with no explicit racial data in the algorithm.
Obermeyer et al., Science 2019 — Landmark empirical demonstration of proxy variable bias operating invisibly through cost-based health algorithms; foundational paper for the medical AI fairness field.
Boston University, 2024 — Explains why removing sensitive attributes fails; bias mitigation must address societal conditions producing biased data.
JAMA Network Open, 2023 — Governance principles framework for proxy variable bias through the full algorithm lifecycle: design, validation, and monitoring.
Multimodal-Specific Adversarial & Reasoning Safety
Multimodal architectures introduce unique attack surfaces absent in unimodal systems — visual modalities can bypass text safety alignment, and benign input combinations can trigger dangerous reasoning chains.
Harmful instructions are encoded within medical images (X-rays, dermoscopy images) — not text — bypassing text-only safety alignment to elicit refused dangerous outputs: lethal drug doses, prohibited procedures.
3MAD dataset + MCM attack (Huang et al., 2024): Multimodal Cross-optimization simultaneously adversarializes image perturbations and text tokens. Attack Success Rate far exceeds pure-text or pure-image jailbreaks. Medical models' text safety guardrails are rendered nearly ineffective when the visual modality is weaponized — a fundamental architecture vulnerability.
Huang et al., 2024 — First systematic cross-modality jailbreak against medical MLLMs; introduces 3MAD attack dataset and MCM optimization algorithm.
Qi et al., AAAI 2024 — Establishes visual modality as an effective safety-bypass vector; theoretical and empirical foundation for medical cross-modal jailbreak research.
ACL 2025 — Visual-textual linkage attacks achieve high success rates without exposing obviously malicious content; stealthy enough to evade manual review.
2024 — Classifies cross-modality jailbreaks as the highest-severity medical AI vulnerability; proposes standardized red-team evaluation protocols.
Image-text mismatches — whether intentional or from EHR errors and image misfiling — exploit cross-modal reasoning failures, causing models to generate hallucinated pathology "confirming" textual expectations rather than image reality.
A healthy lung X-ray + "patient suspected of lung cancer, awaiting imaging confirmation" text causes multiple medical models to fabricate nodule descriptions confirming the text's expectation — directly contradicting the image. In real deployment, EHR system errors or archive misidentification could inadvertently trigger this exact scenario.
Huang et al., 2024 — Simultaneously defines mismatched attacks; models realistic clinical error scenarios (wrong image-patient pairing) as inadvertent attack vectors.
Xia et al., NeurIPS 2024 — Robustness dimension includes cross-modal consistency testing; systematically quantifies Med-VLM failure under mismatched inputs.
Combinations of individually benign inputs trigger dangerous reasoning chains — a blood panel image + pharmacology question may reason into an overdose guide. Single-input safety filters cannot detect this.
Safe Semantics, Unsafe Interpretations: A patient's blood panel image (benign) combined with an academic question about a drug mechanism (benign text) leads the model's chain-of-thought to generate specific toxic dose guidance. Neither input alone triggers any content filter — a fundamental blind spot in current safety architectures.
2024 — First formal introduction of "implicit reasoning safety"; constructs benign-input-combination attack paradigm with medical case studies.
Shayegani et al., 2024 — Splits harmful content across modalities for in-context model reassembly; directly operationalizes implicit reasoning risk.
RAG-augmented medical systems depend on external knowledge bases for accuracy. A single adversarial injection into a semi-open knowledge base can persistently corrupt model outputs without any direct model access.
MedThreatRAG (2025): In a simulated hospital knowledge base environment, a single malicious injection — a pleural effusion chest X-ray labeled "normal" — causes the RAG system to consistently retrieve and apply this incorrect mapping for all subsequent related queries. The attacker never touches model weights or parameters; the knowledge base is the attack surface.
2025 — First systematic poisoning framework for multimodal medical RAG; three attack strategies; single-injection impact quantification.
Xia et al., EMNLP 2024 — Defense-side RAG reliability improvement; implicitly reveals the security-reliability tradeoff in retrieval-augmented medical AI.
Ethics & Liability Risks
Medical multimodal AI introduces fundamental ethical and legal conflicts around informed consent, liability, transparency, and deceptive design — existing frameworks cannot adequately address these.
When AI-assisted diagnosis causes patient harm, no clear legal or ethical framework determines responsibility among the clinician, hospital, AI developer, and regulator — creating accountability vacuums that undermine trust and safety incentives.
A radiologist follows an AI's "low suspicion" output and defers a biopsy; the patient is later diagnosed with early-stage cancer. Three simultaneous unresolved questions: Did the physician adequately supervise the AI? Does the developer's performance specification constitute a medical recommendation? Is the hospital liable for deploying AI without adequate staff training? — All three parties may escape full accountability under current law.
Royal Society Open Science, 2025 — Comprehensive analysis of the accountability gap; redefining legal and professional norms for hybrid human-AI decision-making.
WHO, 2024 — First WHO guidance dedicated to medical LMMs; 40+ governance recommendations with accountability as a core governance principle.
Body & Society, 2025 — Proposes data stewardship as an institutional accountability mechanism; examines operationalizability in medical AI contexts.
Foundation model decisions are opaque to clinicians and patients. Physicians cannot challenge AI outputs meaningfully, nor explain AI-assisted decisions — undermining informed medical practice and the therapeutic relationship.
Qualitative study of healthcare professionals (Yoo et al., 2023): Clinicians' primary concern about medical AI is not its accuracy — it is the inability to understand its reasoning. When AI and clinical judgment conflict, physicians cannot validate either; they face a choice between two black boxes. This is the primary barrier to appropriate trust calibration.
WHO, 2024 — Positions "intelligibility" as a foundational governance principle; establishes explainability as a prerequisite for patient autonomy protection.
PLOS Digital Health, 2025 — Identifies transparency as the most practically challenging ethical dimension; proposes pharmacovigilance-inspired continuous AI transparency auditing.
Patients are largely unaware of AI involvement in their diagnoses, the provenance of training data (potentially including their own records), or AI-specific limitations — making genuine informed consent structurally impossible under current practices.
An NHS survey (2024): over 70% of patients did not know AI was being used to analyze their imaging studies. When informed, many expressed they would have withheld consent if they knew their data was used for commercial AI training — a direct contradiction with existing "implied consent" data practices in many healthcare systems globally.
Royal Society Open Science, 2025 — Redefines informed consent for the AI era; analyzes dynamic consent as an alternative to one-time authorization.
Body & Society, 2025 — Reframes consent as continuous participation rather than one-time authorization; proposes stewardship institutions as mediators.
Medical AI embedded in chatbots and social robots deliberately blurs the AI/human boundary using emotional language and persuasive design — creating false trust that overrides patient judgment and obscures system limitations.
Analysis of commercial medical AI assistant products (2024) found systematic deceptive design patterns: interfaces obscuring the AI/human boundary, emotional language ("I'm deeply concerned about your symptoms") creating false empathy, and "stabilizing" scripts deployed to prevent dissatisfied patients from seeking second opinions — rather than honestly acknowledging AI limitations.
2024 — First systematic taxonomy of deceptive design patterns in medical AI social robots; examines conflicts with medical ethics norms.
WHO, 2024 — Mandates clear AI identity disclosure as a core governance requirement; frames anti-deception as essential to maintaining healthcare system trust.
Socioeconomic Impact Risks
Medical multimodal AI deployment is not only a technical question but a profound distributive one — reshaping healthcare access, transforming the medical workforce, enabling new forms of data extraction, and embedding commercial logic into clinical decisions.
State-of-the-art medical AI deploys first at well-resourced elite institutions, creating a paradox: populations most harmed by algorithmic bias are served by institutions with least AI access.
As of 2024, over 75% of FDA-cleared medical AI algorithms are deployed at the top 20% of large U.S. healthcare institutions. Community hospitals serving majority minority and low-income populations remain largely outside the AI deployment wave — structurally compounding existing healthcare inequality rather than alleviating it.
2024 — Political-economy analysis of how high-income-country AI ecosystems reproduce and deepen historical health inequalities globally.
WHO, 2024 — Positions accessibility and affordability as core equitable deployment principles; proposes international coordination mechanisms for low-income country access.
Medical AI rapidly reaches expert-level performance in imaging-intensive specialties, restructuring training pipelines and eroding foundational clinical skills — with long-term patient safety implications for the next generation of clinicians.
U.S. radiology residency programs have begun reducing high-volume repetitive reading rotations as AI takes over these cases. Paradoxically, this reduces the clinical experience trainees need to effectively supervise AI in complex edge cases — a structural skill-degradation loop where AI adoption creates the very supervisory capability deficit it requires to be safe.
Multi-institution, 2024 — Discusses AI's systemic impact on medical professional identity and training; long-term patient safety implications of reduced foundational skill development.
PLOS Digital Health, 2025 — Addresses transformation of clinician roles to "AI supervisors"; analyzes new training requirements and competency framework changes.
Large tech companies collect patient data from low-income countries under "global health progress" framing, train commercial AI, and sell products at prices those contributing institutions cannot afford — a new form of knowledge dispossession.
Multiple African nations have documented cases where hospital imaging data was incorporated into global medical AI databases (diabetic retinopathy, tuberculosis detection), used to train commercial products now licensed at prices completely beyond the economic capacity of the contributing hospitals — making data contributors systematically excluded from the technology they enabled.
2024 — Introduces data colonialism framing; reveals power asymmetries in data ownership and technology access through mixed-methods African case studies.
PMC, 2021 — Proposes open science as a counter to data colonialism; advocates globally equitable data co-ownership preserving data sovereignty.
Commercial AI optimization logic conflicts with medical ethics logic. Motivated actors (insurers, pharma companies, competitors) have strong financial incentives to manipulate medical AI systems — making these attacks more persistent and harder to detect than random malicious attacks.
JMIR (2026) explicitly models realistic commercial adversaries: insurers incentivized to train AI toward "low-risk" diagnoses to reduce claim payouts; pharmaceutical companies who may exploit prescribing AI to favor proprietary medications. These attacks are commercially motivated to remain invisible — and existing oversight systems have no mechanisms to detect them.
JMIR, 2026 — Models insurance companies, pharma firms, and competitors as medical AI adversaries; integrates commercial conflict of interest into the security threat framework.
Obermeyer et al., Science 2019 — Canonical case of commercial optimization logic (cost-as-proxy-for-need) producing systematic racial discrimination at scale.
Advances in Consumer Research, 2025 — Analyzes how EU AI Act, FDA framework, and WHO guidelines are classifying commercial medical AI under mandatory high-risk oversight regimes.
Risk Overview
| Category | Key Sub-risks | Primary Harm |
|---|---|---|
| Clinical Safety | Hallucination, unreliable decision support, cross-modal misread, outdated knowledge, over-reliance | Misdiagnosis, delayed treatment, patient injury |
| Privacy & Security | Data leakage, prompt injection, adversarial examples, backdoor attacks | Patient privacy violation, system manipulation |
| Algorithmic Bias | Racial bias, gender bias, geographic inequality, proxy variable bias | Systematic underdiagnosis in marginalized groups |
| Multimodal-Specific | Cross-modal jailbreak, mismatched attacks, implicit reasoning, RAG poisoning | Safety guardrail failure, dangerous content generation |
| Ethics & Liability | Accountability vacuum, black-box opacity, consent, deceptive design | Legal gray zones, patient autonomy erosion |
| Socioeconomic | Technology gap, workforce disruption, data colonialism, commercial conflict | Health inequality amplification, systemic exploitation |
References
All cited papers — click titles to access the source where available.