What makes writing ‘good’? A question dear to critics and linguists has been put to reasoning-enabled language models. An academic study investigated how DeepSeek – and, exploratively, Qwen QwQ – form aesthetic judgments, extracting their ‘implicit theory’ directly from reasoning traces. The results say a lot about the nature of these evaluations and open concrete scenarios for those deploying on-premises LLMs for text analysis.

Beyond correctness: intentionality and voice

The research built a benchmark of 30 real texts across six quality tiers, from canonical literature to anonymous forum posts. Over five replications, DeepSeek achieved a mean accuracy of 79.3% in classifying the tiers. The reasoning traces reveal a stable hierarchy: the model rewards expressive intentionality, depth, and a distinctive voice, while overlooking formal correctness. When researchers tested style-matched but unrecognizable passages (purpose-written pastiches), source recognition inflated scores, though the real quality gap between originals and imitations confounds the effect.

Controlled degradation: vocabulary matters little, structure matters a lot

To probe this value scale, the study systematically degraded five canonical passages applying six manipulations: vocabulary simplification, rhythm flattening, imagery removal, voice genericization, structural simplification, and a combination of all. The smallest quality loss came from vocabulary simplification (0.41 points ± 0.46). Loss from structure (2.78) and voice (2.34) is much higher. Combined degradation causes a devastating drop (-5.64), but the effect is subadditive: the sum of individual damages exceeds the joint damage, indicating that the components interact. The check with Qwen QwQ shows the same qualitative pattern, hinting at a cross-model property of current LLMs.

Local inference and structural sensitivity

For organizations evaluating texts in self-hosted settings – whether for automated writing feedback, corporate content quality control, or document auditing – the results demand two considerations. First, the priority given to structure and voice calls for models capable of nuanced reasoning, not simple n-gram classifiers. DeepSeek and Qwen, both runnable on-premises with appropriate hardware attention, embody this direction.

Second, and more delicate: quantization. If evaluation depends on long-range structural features, aggressive quantization (INT8 or lower) might degrade precisely the ability to model complex textual dependencies, while the impact on lexical aspects would be more tolerated. This is not a condemnation but a design constraint: those fine-tuning for stylistic analysis on local infrastructure must test the effect of model format on judgment metrics, not just on perplexity.

Finally, the data sovereignty issue. Evaluating thousands of internal documents with a cloud LLM means exposing potentially sensitive content. Self-hosting eliminates the risk but shifts the cost to correct VRAM sizing and model choice. AI-RADAR has explored in other analyses the TCO of local configurations for inference: a node with one or more consumer GPUs may suffice for 7–13B parameter models, but requirements rise if fine-tuning demands FP16 precision and long contexts to capture extended document structure. The trade-off is between judgment fidelity and hardware cost.

The research offers no ready-made solutions but signals a direction: computational aesthetics is a tough proving ground for LLMs in local production. And the models themselves, with their implicit preferences, are already writing the rules of this game.