A pathology slide can contain tens of thousands of cellular details. Until now, the best AI models analyzed those details by classifying them into predefined categories: carcinoma, benign, suspicious. PRISM2, from Paige and Microsoft, breaks this mold: it doesn’t just classify—it generates text that answers diagnostic questions, trained on clinical dialogue extracted from 685,507 real pathology reports from Memorial Sloan Kettering Cancer Center.

The architecture is a two-phase operation that bets on an ambitious technical trade-off. In the first phase, a perceiver-based encoder aggregates representations of individual tissue tiles into a single vector representing the entire slide. Two loss functions work simultaneously: a contrastive one, driven by BioGPT text embeddings, aligns the slide representation with report language; an autoregressive one, using Phi-3 Mini, forces the encoder to generate text directly. The idea is that contrastive training alone produces embeddings useful for retrieval but weak for generation, while autoregressive training can memorize superficial text patterns with little transfer to the visual domain. Running both losses against the same encoder output is PRISM2’s architectural bet.

The second phase freezes the encoder and puts the language model (4 billion parameters) to work on clinical dialogue, teaching it pathology reporting conventions without touching the visual mechanics. From this structure emerge two types of embeddings: a “base” one, straight from the encoder, which transfers well to tasks like biomarker prediction; and a “diagnostic” one, extracted from the language model’s hidden state after it has processed both the slide latents and the textual prompt, optimized specifically for cancer detection and subtyping. Survival tasks get an even more specialized, third, separately fine-tuned embedding.

The numbers are strong. PRISM2 matches or exceeds the balanced accuracy of clinical-grade products calibrated for prostate and breast cancer, and outperforms Paige BLN on breast lymph nodes without any further task-specific training. In pan-cancer detection, the diagnostic embedding’s AUC reaches 0.967, against 0.956 for the base embedding and 0.947 for the earlier PRISM model. An ablation study shows that adding dialogue templates lifts prompt-based inference from 0.498 to 0.653 balanced accuracy, and about half of PRISM2’s overall diagnostic improvement is attributed to the scale increase of the question-answer dataset, which is 3.5 times larger than the one used for the original PRISM.

But the way that dataset was built raises some questions. The 685,507 reports were turned into question-answer pairs by GPT-4o, a synthetic-data pipeline that needed independent quality checking. A pathologist reviewed 50 specimens: errors in open-ended and multiple-choice questions landed at 3%, but rose to 8% for diagnostic summaries and 18% for yes/no questions, often irrelevant or inaccurate. PRISM2’s own QA errors sit at 7–11%, driven mainly by hallucination and omission rather than factual contradiction of the slide.

The real friction point, however, is reproducibility. Model weights are available on Hugging Face, but training and inference pipelines depend on proprietary Paige and Microsoft infrastructure. This means an outside team can run inference against the released weights but cannot recreate the full workflow—a gap that weighs on anyone wanting to bring these models into on-premise environments to keep patient data under control. Even the generation of the synthetic dataset requires GPT-4o, a closed model, adding another external dependency. For those evaluating on-premise deployment of medical AI, AI-RADAR offers analytical frameworks at /llm-onpremise to weigh such trade-offs.

On top of these limitations come technical gaps that an ML team cannot ignore: the model has no position encoding across tiles, so it cannot reason about the spatial arrangement of structures within a slide, and every scan was performed at a single fixed resolution of 0.5 microns per pixel, using a single scanner type (MSK’s). Extending PRISM2 to variable magnifications or different centers will require independent validation. The path toward truly sovereign, reproducible pathology AI is still uphill, but this work confirms that clinical language can become an effective supervisor when the scale is large enough. Doing so without chaining oneself to proprietary infrastructure is the next challenge.