The most interesting number in the study is not the 64.9% accuracy achieved by Rationale Labeling against the 57.8% of the expert-revised codebook. It is that the result comes without asking for more compute or larger models: it comes from asking experts to work only on cases where multiple LLMs disagree. On thousands of tutoring-session transcripts, the researchers compared three ways of integrating human feedback into codebook revision: editing model-generated revisions driven by cross-model disagreement, answering questions about the disagreements, and labeling disagreement cases with rationales. The last method produced the highest labeling accuracy and even outperformed the codebook manually revised by experts. The best Question Answering variant reached 60.5%, still above the expert baseline.

What is at stake here is not just annotation. A robust codebook takes months because every rule must be checked against real cases, and ambiguous cases emerge slowly. If model disagreement can surface them early, expert time becomes a resource to be allocated by a criterion rather than spread evenly. For teams running large-scale annotation pipelines, this shifts the bottleneck: the model is no longer the limit, but the ability to select the right examples for a human to review.

What changes for local deployments

For those evaluating self-hosted stacks, the mechanism has a structural consequence. Disagreement among multiple LLMs is a cheap signal that requires neither external APIs nor pre-existing labels: it only needs two or more local models, not necessarily state-of-the-art, applied to the same documents. In an on-premise scenario, this means domain-specific codebooks can be built without moving data outside the corporate perimeter and without waiting for a cloud provider to update its annotation service. The cost, however, shifts elsewhere: you need VRAM to run multiple models in parallel, or a pipeline that runs them sequentially with longer inference times. Quantization can reduce memory usage, but it introduces another variable in model behavior and therefore in the quality of the disagreement signal.

The benefit is not uniform. Organizations working with highly specific domains — legal documents, clinical reports, technical support, educational content — gain more because the generic codebook shipped with open models is often far from the real vocabulary. Teams managing generic annotation with a single proprietary model may find the method less useful, because cross-model disagreement is unavailable or treated as noise to be removed. This is where the study signals a shift in incentives: label quality no longer depends only on model scale, but on the architecture of the human review workflow. And that favors those who control the models and the data, not those selling closed solutions.

One point remains open: 64.9% is a result on a specific domain, measured as accuracy against expert labels. Generalizing will require tests on other text types and larger codebooks. But the principle is clear: disagreement is not an error to suppress, it is a resource for deciding where to invest human attention. In an industry obsessed with GPU power, it is a reminder that the bottleneck can be organizational, not computational.