In medicine, a model that is confidently wrong can do more harm than one that admits doubt. A controlled, psychophysics-inspired benchmark tried to measure exactly this behavior on gpt-4.1-nano: not just how many diagnoses are correct, but whether stated confidence tracks evidence quality.

The test used 45 synthetic vignettes simulating probable Alzheimer-type neurocognitive disorder (AT-NCD) or depression-related cognitive impairment (DRCI). Each vignette was presented under three prompt variants, for a total of 135 trials. Diagnostic accuracy was 93.5%, mean confidence was 78.4%, and AUROC2 was 0.876. Those numbers look solid, but the interesting part lies elsewhere.

The model's confidence increased as evidence moved away from the diagnostic boundary and decreased when information was missing. After adjusting for evidence strength and prompt format, confidence remained higher on correct trials than on incorrect ones. This points to partial metacognitive sensitivity: the signal is not globally uninformative. But errors clustered in moderate, conflicting AT-NCD cases, where the model shifted toward DRCI and kept more confidence than empirical accuracy justified.

The issue is not that a model makes mistakes on difficult cases; it is that it does so with excess confidence exactly there. For anyone evaluating a clinical LLM, this changes the kind of testing required. Average accuracy on a benchmark hides localized calibration failures, and those failures only emerge through tests built to push the boundary between categories. The study also suggests measuring confidence quality directly rather than inferring it from benchmark accuracy or general model capability.

In an on-premise setting, this approach has immediate operational value. Healthcare organizations running self-hosted models can replicate calibration tests on internal data, without sending clinical records to external services and with the ability to freeze the model version during evaluation. That is not a minor detail: calibration can shift between versions, after fine-tuning, or with different prompts, so a one-time test is not enough. What is needed is continuous monitoring, embedded in the validation pipeline, with metrics that separate accuracy from confidence quality across case subgroups.

Who benefits from this view? Teams building local validation stacks, because they can turn such a benchmark into a regression test for every model update. Who risks losing? Vendors offering accuracy as sufficient proof for clinical use, and hospitals that adopt models without checking calibration on their own case mix. Structurally, the signal is clear: in high-stakes domains, evaluation is moving from abstract capability to model behavior during inference on realistic data.

For those evaluating on-premise deployment, AI-RADAR offers analytical frameworks at /llm-onpremise to weigh these trade-offs, but calibration remains a use-case-specific exercise. The next step is not asking whether a medical LLM is accurate enough, but whether it knows when to stop being so.