What physicians should know about how AI works

A colleague asks a chatbot about a rare drug interaction. The answer arrives in seconds: confident, well structured, complete with a citation to a 2021 study in a respected journal. But: the study does not actually exist. Nothing malfunctioned: the system did exactly what it was built to do.

For physicians using AI for clinical decision-making, it is crucial to understand risks and limitations of AI tools and where they come from. This article is a starting point for understanding these risks and how to navigate them. Data protection while using AI is a separate topic and partly covered in this article.

⏱️ Reading time: approx. 7 minutes

🧠 How a large language model actually works

At its very core, a large language model (LLM, the technology behind ChatGPT and most AI tools) generates text by answering one question, over and over: given everything written so far, which word is a likely continuation?

To do this, the model splits text into tokens - words or parts of words - and positions them on a huge map with many dimensions called the embedding space. Terms that appear in similar contexts in the training data sit closer together on the map: “dyspnea” ends up near “shortness of breath”, and in the neighborhood of “heart failure”. The distances here are what the model works with. It calculates at every step a probability for each possible next token and then picks one of the likely candidates, not always the top one. This is why the same question can produce different answers each time you ask.

👉 Because the model’s objective is to calculate which token is more likely to come next, it doesn’t work explicitly to ascertain that the resulting statement is correct.

👉 Because the model is trained to predict what a human would plausibly write next, it reproduces the fluency of its training data. [1] A wrong answer therefore sounds exactly like a correct one.

An AI-generated simplified visualisation of the embedding space.

🚨 Hallucination: a property of the method

The LLM design can lead to “hallucinations”, meaning confident statements that happen to be false. The model produces it the same way it produces everything else: by assembling the statistically most plausible continuation. Especially in fields where it has little training data, it fills the gap with tokens that look right. Fabricated output usually looks exactly like real output. Invented sources can have plausible authors, journals and page numbers [1].

Gaps in academic research are one danger field, because academic publications follow rigid, predictable patterns that are easy to imitate for LLMs. In an analysis of ChatGPT-generated short papers, 47% of the references were fabricated (115 references across 30 papers), a further 46% were real but contained errors, and only 7% were both authentic and accurate [2]. A second study on multidisciplinary topics found that 55% of citations produced by GPT-3.5 were invented. GPT-4 brought this down to 18% [3].

👉 Physicians should critically question suggestions and ideas from AI models and cross-check studies and sources.

⚠️ Sycophancy: The model wants to agree with you

The goal of LLMs is to be helpful and they tend to agree with whatever the user suggests. Researchers call this behavior sycophancy. This happens because LLMs are trained and fine-tuned on positive feedback by human evaluators that unsurprisingly prefer nice, polite and helpful responses.

A second effect works at the level of the individual prompt. The model does not evaluate your prompt from the outside - it continues it. The model may accept the assumptions in a prompt rather than testing them. A stated suspicion can therefore shape the response as if it were a premise, particularly where the prompt is framed confidently.

A 2025 US study from Mass General Brigham tested this directly. Researchers asked five LLMs to write patient advisories based on deliberately illogical premises, for example that “Paracetamol” had new side effects and patients should switch to acetaminophen (the same drug). The models complied frequently, even though they demonstrably “knew” the two names refer to the same substance [4].

👉 Especially at the beginning of a longer conversation, every prompt should be neutral and not already suggest a suspicion or idea.

👉 When possible, AI tools should be specifically configured in their instructions to critically question assumptions and suggestions inputted by the user. This is often possible in the relevant settings of common tools.

📊 The patient in the training data is not your patient

An LLM absorbs the patterns of its training data, including the demographic and social patterns of the healthcare systems that produced it. These are often not transferable to other world regions or contexts, particularly for specific patient groups.

Two US findings show the range of the problem. The first does not involve a language model at all: a widely used commercial care-management algorithm used healthcare spending as a proxy for medical need. It systematically underestimated illness in Black patients, because less money was historically spent on their care at the same level of need [5]. This mechanism is the same across model types. In another finding, when GPT-4 was asked to generate teaching vignettes and differential diagnoses, it exaggerated known demographic prevalence differences and produced differentials that stereotyped by race, ethnicity and gender. For some conditions it generated female patients almost exclusively [6].

Bias in AI output is hard to notice case by case, because each individual answer looks reasonable. It becomes visible only in aggregate.

👉 Findings and responses by AI should be treated as a new hypothesis to check against better-matched data, not as a finding that applies to your patient.

🧠Automation bias: The risk on your side of the screen

Automation bias is a well-documented human tendency to over-rely on automated suggestions, described well before language models existed in a 2012 systematic review of rule-based clinical decision support systems [7]. Under time pressure and with many competing tasks, verifying a suggestion costs more mental energy than accepting it. This behaviour is called cognitive offloading. Humans tend to perceive the fluent and confident language of LLMs as competent, even if the output is not correct. AI can also produce step-by-step explanations that sound logical, but end up at the wrong conclusion.

A German experimental study in Radiology measured how strong the effect is on this perceptual reading task: when a purported AI system suggested the correct Breast Imaging Reporting and Data System (BI-RADS) category for a mammogram, inexperienced radiologists (11 out of 27 participants) were right in about 80% of cases. When the same system suggested a wrong category, their accuracy dropped below 20%. Very experienced radiologists - a group of five in this study - fell from 82% to 45.5% [8].

👉 Physicians should not immediately believe any suggestions by an AI and instead cross-check them or, if possible, re-think the suggestion at a later point in time to avoid falling to automation bias.

📌 Context window: where your input gets lost

Every word you type into an AI model gets converted into tokens, and the model has to relate each token to every other token in the input to understand context and formulate a response. This effort grows disproportionately as the input gets longer. To keep costs in check, every model is trained up to a fixed maximum input length: this ceiling is called the context window. Everything you paste in has to fit, and even when it fits, it is not weighted evenly.

US Research on long inputs found a consistent pattern: models attend most reliably to information at the very beginning and very end of the input, and perform worst on information buried in the middle, the so-called “lost in the middle” effect [9].

For clinical work, this danger is relevant when inputting long documents or a lot of data. In a big medical record, a critical allergy noted somewhere in the middle risks being overlooked for its final output.

👉 Physicians should not assume that a long document or a large amount of data has been processed reliably. Critical findings should be stated explicitly at the beginning or ending of the prompt and checked for in the output, rather than assumed present because the document was attached.

💡 Most important things at a glance

The following are editorial recommendations derived from the evidence above, not study findings themselves.

The risks described here are not malfunctions. They follow from how language models work. Fluency, confidence and correct formatting are properties of the generator, not evidence about the content.

• The output can be wrong in ways that look right. Fabricated references, invented details and confidently stated errors are produced by the same mechanism as correct output, and they carry the same surface markers of quality [2, 3].

• The model adapts to the user, not to the evidence. A suspicion stated in the prompt is treated as given rather than tested, and models trained on human approval tend towards agreement [4].

• Distortions in the training data reappear in the output. Demographic and system-specific patterns are reproduced and sometimes amplified, and they are hard to detect in any single answer because each one looks reasonable on its own [5, 6].

• The main risk sits on the human side of the screen. Under time pressure, accepting a suggestion costs less effort than verifying it - and a wrong suggestion measurably degrades the judgment of experienced clinicians [7, 8]. Long inputs add to this: information in the middle of a document receives the least attention from the model [9].

Together these effects mean output from common AI chatbots cannot be treated the way a specialist’s report or a guideline is treated. Professional accountability for the decision remains with the physician who signs it.

📚 Sources

1. Ouyang et al., Training language models to follow instructions with human feedback, NeurIPS 2022 (arXiv:2203.02155), https://doi.org/10.48550/arXiv.2203.02155.

2. Bhattacharyya M et al. High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content. Cureus. 2023;15(5):e39238. https://doi.org/10.7759/cureus.39238

3. Walters WH, Wilder EI. Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports. 2023;13:14045. https://doi.org/10.1038/s41598-023-41032-5

4. Chen S et al. When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior. npj Digital Medicine. 2025;8:605. https://doi.org/10.1038/s41746-025-02008-z

5. Obermeyer Z et al. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447–453. https://doi.org/10.1126/science.aax2342

6. Zack T et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digital Health. 2024;6(1):e12–e22. https://doi.org/10.1016/S2589-7500(23)00225-X

7. Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2012;19(1):121–127. https://doi.org/10.1136/amiajnl-2011-000089

8. Dratsch T et al. Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology. 2023;307(4):e222176. https://doi.org/10.1148/radiol.222176

9. Liu NF et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. 2024;12:157–173. https://doi.org/10.1162/tacl_a_00638