In theory, artificial intelligence in medicine should lighten the load on professionals, not eliminate them. Two stories that emerged this week in the United States flip that vision on its head. In New York, a group of nurses claims they have been replaced by a decision-support software. In Minnesota, a former senior executive at the Mayo Clinic – one of the world’s most respected healthcare institutions – publicly states that the AI introduced into the facility was not safe and did not deserve trust.

The New York case is not isolated. Marilyn Shuler, a nurse with thirty-nine years of experience, has recounted how her expertise in reading exams has been rendered superfluous by an algorithm that now performs the same evaluations without direct human oversight. This is not assistance – it is a full functional replacement. In Minnesota, the former Mayo Clinic leader – whose name has not yet been officially released – raised alarm about the lack of independent testing and the absence of rigorous clinical validation before deployment.

These episodes signal a dangerous gap between vendor rhetoric and hospital practice. Who benefits in the short term? Technology suppliers, pushed to place off-the-shelf systems, and healthcare administrators who see automation as a way to cut staffing costs. Those who lose out are patient safety and the professionalism of clinicians, reduced to passive overseers or entirely removed from the decision-making process.

The underlying issue is not AI itself, but the speed at which it is being introduced into high-risk contexts without adequate checks. In medical devices, stringent certification procedures exist; for AI software, by contrast, the loopholes are wider. Many algorithms are implemented as support tools that, in practice, operate autonomously, thus bypassing the regulatory scrutiny required for actual medical devices. The result is a regulatory short circuit: the hospital can claim to have introduced a “suggester,” but on the wards that system de facto decides who to discharge, which therapy to prescribe, or when to alert a physician.

At a structural level, this raises questions of data sovereignty and audit. Who monitors the algorithm’s performance after installation? Often, nobody. The black boxes of AI make it difficult to reconstruct decisions, and in the absence of transparent logs or continuous monitoring infrastructure, errors can go unnoticed for months. This scenario is particularly critical when the software runs on external clouds, outside the hospital’s security perimeter, heightening risks of opacity and privacy breaches.

For those evaluating on-premises deployment, the case poses a central question: if AI were run on local servers, with full audit logs and a clear governance model, would it be easier to detect malfunctions and assign accountability? It is not an automatic fix, but it shifts the balance toward effective control by the healthcare institution. AI-RADAR has long tracked the evolution of local inference frameworks in regulated environments, where software certification and pipeline transparency become discriminating factors.

These two American stories are not isolated incidents: they are the tip of an iceberg. The accelerating adoption of AI in healthcare, fueled by billions in investment and promises of efficiency, is outstripping regulators’ ability to keep pace. Unless a shift toward independent validation is enforced, the next case of algorithmic malpractice could turn into a crisis of confidence for the entire sector.