In a randomised clinical trial published in JAMA Network Open, fifty physicians worked through clinical cases with and without access to a large language model. Having the model in front of them did not improve their diagnostic reasoning: 76% against 74%, a two-point difference that does not reach statistical significance. The same model, working alone, scored 16 points above the physicians. That double result — the machine wins alone but adds nothing to the professional — is the most interesting thing this field has produced, and it defines the real problem: it is not about model capability, it is about how the model fits into a person’s work. This article walks through the five measurements that support it.
Each row names the study and the year. Bear in mind that every single one of them evaluates written cases rather than actual patients.
| Finding | Result | Study and year |
|---|---|---|
| Physicians with the model against physicians without it | 76% against 74%; 2-point difference (CI: −4 to 8; p = 0.60) | Goh et al., JAMA Netw Open (2024) |
| The model working alone | 16 points above physicians with conventional resources | Goh et al., JAMA Netw Open (2024) |
| Failure generating a differential diagnosis | Rate above 0.80 in all 21 models evaluated | Rao et al., JAMA Netw Open (2026) |
| Failure giving the final diagnosis | Below 0.40 in those same models | Rao et al., JAMA Netw Open (2026) |
| Anchoring bias: ranking the suggested diagnosis first | 55.6% for the models, 21.2% residents, 10.0% attendings | Sheppert et al., IJMI (2026) |
| Diagnostic accuracy across 30 studies and 4,762 cases | Between 25% and 97.8%, with high risk of bias | Shan et al., JMIR Med Inform (2025) |
| Spread between models on 148 neurosurgery cases | From 74% down to 36% depending on the model | Warrier et al., J Neurosurg (2026) |
• Where exactly the model fails: not on the diagnosis, but on the differential.
• Why the meta-analytic range runs from 25% to 97.8%.
Ethan Goh, Robert Gallo along with an extensive team released a single-blind randomized clinical trial via JAMA Network Open during 2024. Fifty practitioners participated — comprising 26 attending physicians and 24 residents possessing a median practice tenure of three years — hailing from emergency, internal, and family medicine disciplines, who were allotted 60 minutes to evaluate as many as six medical scenarios. Assignment by randomization determined whether these clinicians could utilize the language model alongside standard reference materials.
Scoring used a validated rubric covering differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic steps, graded by blinded expert consensus.
The primary result: a median of 76% per case in the model group and 74% in the group without it, an adjusted difference of two percentage points (95% CI: −4 to 8; p = 0.60). Nor was there a significant difference in time per case.
The authors added a secondary comparison: what happened if the model solved the cases with no human involvement. The model alone scored 16 percentage points above the physician group using conventional resources (95% CI: 2 to 30; p = 0.03).
Their conclusion is the right reading of the whole set: technology and workforce development are needed to realise the potential of physician-AI collaboration. The bottleneck is not model capability but the interaction.
| Study | What it measured | What it found | What it cannot support |
|---|---|---|---|
| Goh et al., JAMA Netw Open (2024) | Randomised trial with 50 physicians and clinical cases | The model did not improve the doctor; alone, it beat both groups | Written cases, not real patients |
| Rao et al., JAMA Netw Open (2026) | 21 models across 29 cases, 16,254 responses | Good at final diagnosis, poor at the differential | The graders were medical students |
| Sheppert et al., IJMI (2026) | Diagnostic anchoring in 8 models and 25 physicians | Models anchor far more than people | A forced-choice task, not open clinical practice |
| Shan et al., JMIR Med Inform (2025) | Meta-analysis of 30 studies, 19 models, 4,762 cases | Accuracy between 25% and 97.8% | High risk of bias: cases already had known diagnoses |
| Warrier et al., J Neurosurg (2026) | 148 neurosurgery cases across five models | From ChatGPT-4’s 74% to Gemini’s 36% | Textbook cases, with queries run in May 2024 |
In 2026, Arya Rao and co-workers released the most comprehensive assessment to date in JAMA Network Open: 21 general-purpose models, incorporating the newest iterations, evaluated over 29 standardised clinical cases, with 16,254 outputs graded three times across the sequential phases of the traditional clinical pathway.
The finding that matters is not the overall score but its breakdown. Failure rates exceeded 0.80 for differential diagnosis in every model, while for final diagnosis they stayed below 0.40.
Translated: these systems hit the correct diagnosis reasonably often, and they are bad at generating the list of alternatives that must be ruled out first. That list is precisely the clinical work, and what protects the patient when the obvious diagnosis turns out to be the wrong one.
The authors’ conclusion is explicit: despite version-based improvements, off-the-shelf models have not yet achieved the intelligence required for safe deployment.
Anchoring is a classic error of clinical reasoning: sticking with the first hypothesis suggested. Alexander Sheppert and colleagues measured it in 2026 in the International Journal of Medical Informatics with a clean design: nine pairs of internal medicine cases, each pair with one version containing a patient statement suggesting a plausible but unlikely diagnosis and one matched version without it.
| Who is responding | Ranks the suggested diagnosis first | Includes it in the top five |
|---|---|---|
| Language models (8 evaluated) | 55.6% (40 of 72) | 97.2% |
| Internal medicine residents (20) | 21.2% (17 of 80) | 67.5% |
| Attending physicians (5) | 10.0% (2 of 20) | 65.0% |
| Mean anchoring-induced shift | Models 2.29; residents 0.88; attendings 0.87 | Significant difference (p = 0.008) |
| Relative odds against residents | OR 3.94 (95% CI: 2.32 to 7.33; p < 0.001) | Against attendings, OR 4.07 |
The pattern is consistent and cuts against the usual intuition. AI is often presented as the antidote to clinicians’ cognitive biases; on this particular measurement, the models let themselves be anchored four to five times more often than an attending physician. And the difference between residents and attendings — less anchoring with more experience — is the expected one.
Guxue Shan and colleagues published in JMIR Medical Informatics in 2025 the most complete systematic review with meta-analysis to date: 30 studies, 19 different models and 4,762 cases, with risk of bias assessed using the PROBAST tool.
The accuracy range for the primary diagnosis in each study’s best model runs from 25% to 97.8%, and for triage classification from 66.5% to 98%. It is not that some models are better than others: it is that the difficulty of the cases chosen changes the result completely.
The authors also flag the limitation that runs through this entire field: most studies carry a high risk of bias, and the main cause is the use of cases whose diagnosis is already known. Their conclusion is that the accuracy of these systems still falls short of clinical professionals.
The magnitude of variation between models is well illustrated by the work of Akshay Warrier and colleagues in the Journal of Neurosurgery, using 148 clinical cases from a neurosurgery manual: ChatGPT-4 got 74% right, AtlasGPT 63%, ChatGPT-3.5 53%, Copilot 48% and Gemini 36%. Most errors came from failing to incorporate a key piece of information into the reasoning, usually imaging.
Ordered by the strength of the evidence behind it:
• Do not assume that giving the tool to the clinician improves diagnosis. The only randomised trial available found no improvement: two points of difference, not significant.
• Do not introduce a hypothesis into the query. Models ranked the suggested diagnosis first in 55.6% of cases, against 10% for attendings.
On written clinical cases, a model working alone can score higher. In the randomised trial published in JAMA Network Open in 2024, the model on its own scored 16 percentage points above the physician group with conventional resources (95% CI: 2 to 30; p = 0.03). That is not the same as diagnosing a real patient, where there is examination, testing and the course of the illness.
The solitary randomised trial at hand indicates no notable difference. With the model, the median diagnostic reasoning score reached 76%, compared to 74% in its absence, yielding an adjusted variance of two percentage points alongside a 95% confidence interval spanning from −4 to 8 points (p = 0.60). Furthermore, the duration required per case experienced no meaningful decrease.
On the differential diagnosis, not the final one. In the evaluation of 21 models across 29 cases and 16,254 responses, failure rates exceeded 0.80 for generating the differential diagnosis in every model, while for the final diagnosis they stayed below 0.40. That is, they get the answer right and do not build the set of alternatives well.
On diagnostic anchoring, considerably more biased. In the 2026 forced-choice experiment, models ranked the patient-suggested diagnosis first in 55.6% of responses, against 21.2% for residents and 10.0% for attendings, with an odds ratio of around 4 against both groups.
Because case difficulty changes the result entirely. The meta-analysis of 30 studies with 19 models and 4,762 cases puts primary diagnostic accuracy between 25% and 97.8%, and flags a high risk of bias in most of the work because cases with already known diagnoses are used.
• Whether anchoring can be corrected. The study documents it in a closed task and expressly calls for replication in open clinical settings.
1. Goh, E., Gallo, R. et al. Large Language Model Influence on Diagnostic Reasoning. JAMA Network Open, 2024. DOI: 10.1001/jamanetworkopen.2024.40969
2. Rao, A. S., Esmail, K. et al. Large Language Model Performance and Clinical Reasoning Tasks. JAMA Network Open, 2026. DOI: 10.1001/jamanetworkopen.2026.4003
3. Sheppert, A., Shen, C. et al. Large language models exhibit greater diagnostic anchoring than physicians in a forced-choice vignette study. International Journal of Medical Informatics, 2026. DOI: 10.1016/j.ijmedinf.2026.106550
4. Shan, G., Chen, X. et al. Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis. JMIR Medical Informatics, 2025. DOI: 10.2196/64963
5. Warrier, A., Levin-Carrión, Y. et al. Comparative diagnostic capability of large language models in neurosurgery. Journal of Neurosurgery, 2026. DOI: 10.3171/2025.9.jns25846
This content is for information only and does not constitute medical advice. The figures cited come from studies carried out on written clinical cases, not on real patients, and none of them describes how these systems behave in clinical practice.
Data and technology company Experian has released RECLAIM: Stories of Passion and Perseverance, a short…
Namibia Breweries Limited (NBL), WWF Namibia and Global Water Challenge have signed an agreement to…
Winning UNESCO status is almost always defended with a tourism argument. It is the argument…
James Gunn and Peter Safran will step down from their roles as co-chairs and co-CEOs…
Introduction: A Century of Industrial TransformationOver the past hundred years, industrial activity has expanded at…
A GP prescribing a painting class instead of an anxiolytic is a powerful image, and…