A single LLM can detect a student's error, recognize the misconception behind it, and correct it in a multi-turn dialogue? According to the MisconceptDialogue framework, the answer depends first on how you measure success. The researchers built a fully automated pipeline with five open-weight LLM tutors from 7B to 14B, interacting with simulated students in more than 5,000 conversations. On the same dialogues, the exact-match scoring typical of math NLP benchmarks reports a Remediation Success Rate of 42.1%, while a human-calibrated LLM judge reaches 81.3%. That is a 39.2 percentage point gap, larger than the combined effect of tutor model, teaching strategy, and supplied diagnostic information.

This is not an artifact of a single automated judge: four independent LLM judges and 200 human annotations converge within a few points, with kappa up to 0.91. The issue, then, is not judge reliability. It is that the chosen criterion can nearly double the measured value on the exact same behavior. Exact match, often used as a lower-bound stress test, is not merely strict: it produces a reading so different that it changes decisions about models and strategies.

The second result is even more uncomfortable. When the tutor is given the correct diagnosis or a deliberately swapped one, the score barely changes. In the tested diagnosis-presentation format, these tutors show little evidence of reliably conditioning their responses on the diagnostic information they receive. In practice, the supposed ability to use diagnosis to guide remediation appears weak. The finding shifts attention from 'is the model good at math?' to 'does the model know how to use what it is told during the dialogue?'.

For those evaluating on-premise deployment of LLM tutors, the structural lesson is that evaluation is not separate from the stack: it is part of the system. An open-weight model from 7B to 14B used in a self-hosted educational service cannot be chosen on static benchmarks alone, because the task is multi-turn and requires measuring remediation, not just the final answer. The framework separates error detection, diagnosis, and intervention, and shows that the number changes depending on how the outcome is judged. It calls for an evaluation pipeline with multiple criteria and multiple judges, anchored to human decisions.

There is also a second-order effect on incentives. If exact match remains the dominant criterion, model developers or selectors will tend to optimize lexical overlap with expected answers, not the ability to correct reasoning. In an educational context, this rewards tutors that guess the shape of the solution but not the process. The difference between 42.1% and 81.3% is not merely statistical: it can discard an effective system or pass a pedagogically weak one, with the same model and dialogue.

Judge convergence and human anchoring are not methodological details: they are the only way to make benchmarks decision-grade. In self-hosted setups, each additional LLM judge introduces local inference load and organizational costs, but without this redundancy the number remains captive to the criterion. For those weighing on-premise deployment, there are trade-offs between evaluation cost and measurement reliability; AI-RADAR offers analytical frameworks at /llm-onpremise to read them without reducing them to static benchmarks.

Measurement, in the end, does not only tell you about the model: it tells you about the rules you used to observe it.