Anyone who has tried to integrate a Large Language Model into a real-world application knows that success depends not just on answer accuracy, but on the model's ability to deliver output in the expected format. A new large-scale study highlights just how underestimated this dependency is—potentially making traditional rankings misleading.

Researchers ran over 140,000 generations on OpenRouter, covering seven QA tasks, five wrapper families (the way prompts are packaged, from free text to structured XML or JSON), and four instruct models ranging from 7 to 72 billion parameters. They introduced two complementary metrics: the Format Sensitivity Index (FSI), which measures the accuracy spread across different wrappers, and the Parseability Sensitivity Index (PSI), which does the same for output parseability (the ability to reliably extract the answer automatically). The most striking result? Mean FSI varies more than 30-fold between models, and the main culprit isn't differences in "intelligence" but syntactic compliance failures.

In other words, a model can appear excellent on a benchmark simply because for that specific format it produces easily parseable outputs, while a single extra comma or a slightly different tag breaks the parsing system and performance collapses. Fixed-effects regression analysis confirms that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper. This means that part of the score attributed to the LLM's competence is actually an artifact of successful parsing.

The implication for anyone evaluating models for deployment is clear: a bare accuracy number says nothing about system robustness. In production, especially in on-premise architectures where automation is deep and no human operator is on hand to fix things on the fly, the ability to consistently produce structured outputs is as crucial as content correctness. A model that fluctuates wildly in parseability depending on the prompt is a weak link that can halt entire pipelines.

The study offers practical recommendations: every benchmark should report the variance induced by wrappers and a compliance index, because without these data leaderboards are statistically fragile. This shifts attention from scale alone (how large the model is) toward alignment and fine-tuning quality: a smaller but more "obedient" LLM could be a smarter choice for many enterprise use cases.

Ultimately, if we want Large Language Models to become reliable infrastructure components, we must stop evaluating them as oracles of pure textual truth and start measuring their sensitivity to format. The next leaderboards would do well to include, alongside the usual accuracy score, a Format Sensitivity Index—it could turn out to be the real differentiator between a demo and a product.