The idea that a large language model can judge by itself how difficult a reading comprehension question is appears tempting: it would mean automatically calibrated tests, personalized learning paths, enormous savings. Too bad reality, as a new large-scale study reveals, is more complicated.

Researchers put several LLMs to the test in predicting the difficulty level of items from an extensive Reading and Writing test. They compared prompting strategies and parameters, including encoder-only models and traditional feature-based machine learning approaches. The starkest result? Zero-shot GPT-4.1 with temperature zero achieved a quadratic weighted kappa (QWK) of 0.578, but was outperformed by ConvBERT, a much leaner model, which reached 0.625. Worse, further analysis uncovers a structural flaw: all LLMs struggle to label the hardest items. GPT-5.4, despite its power, systematically tends to underestimate difficulty, as if everything were easy for it.

This behavior has deep roots. By reducing embedding dimensionality, researchers observed that semantic representations of items of different difficulty levels blend together. Linguistic information alone is insufficient: the difficulty perceived by a human depends on cognitive factors that an LLM, trained on statistical correlations, does not capture reliably. As the model becomes more capable, its "self-confidence" grows and the world appears simpler – a bias that risks producing trivializing tests if left uncorrected.

For those developing or evaluating automatic item-generation pipelines in education, the message is clear. In an on-premise setting, where data sovereignty or regulations require keeping everything local, one is often forced to use quantized or less powerful models due to VRAM and compute constraints. If state-of-the-art models already struggle to grasp difficulty, a compressed version or one with a reduced context window could amplify the "everything easy" effect. The TCO of a self-hosted solution isn't measured only in hardware and electricity bills: it includes the cost of human validation, the integration of traditional classifiers (like ConvBERT) to correct estimates, and the risk of deploying educational tools that misjudge students.

The structural implication is that the "one model to rule them all" paradigm starts to crack in domains where psychometric precision matters. The encoder-only model, while lacking the generative fluency of an LLM, here proves better at reading difficulty nuances. This points toward hybrid architectures: a powerful LLM (cloud or on-prem) to generate the text, flanked by a smaller, specialized model for difficulty estimation, all orchestrated in a pipeline that cannot do without human oversight.

Ultimately, if someone is considering entrusting a self-hosted LLM with the creation of entire banks of quizzes calibrated to student levels, they should remember that even the most advanced models see the world through the eyes of someone who already knows everything – and for an educator, that's a problem, not a virtue.