The question is no longer just how good an LLM is, but what we ask it to do. A systematic mapping of 14,767 arXiv preprints published between January 2022 and August 2026—papers that introduce or update evaluation resources—shows that research expectations are shifting toward action, interaction, and professional applications. This is not simply a growth in tests: it is a change in how the scientific community defines success.

This shift is not uniform. The study finds that established and newer design elements coexist, and, more importantly, that model participation in the evaluation process itself evolves asymmetrically. On one side, LLM-based scoring grows in both agent and non-agent groups; on the other, model-generated materials show no sustained increase in recent cohorts. In short, models are increasingly judges, but not consistently authors of the tests.

An asymmetry that changes the weight of benchmarks

This asymmetry has deep consequences for deployment decisions. If benchmarks become more action- and interaction-oriented, measuring the quality of a single response is no longer enough: it becomes necessary to assess how a model acts in context, how it interacts with tools, and how it respects operational constraints. For on-premises or self-hosted environments, where control over the inference pipeline is a requirement, this evolution shifts attention from public scores to local tests that include real interaction scenarios.

But the most delicate point concerns the independence of evaluation. When an LLM assigns scores, the verdict tends to reflect the preferences and blind spots of the judging model. If model-generated materials do not grow durably, the risk is that benchmarks remain anchored to categories and formats defined by an earlier generation of models, while evaluation becomes automated around standards that reward already-known response styles. For the industry, this means the benchmark race can reward those who optimize for automatic judges rather than those who build real capability. Vendors have an incentive to shape their systems around the implicit criteria of LLM evaluators, and this makes rankings less useful as an independent signal.

For organizations evaluating on-premises deployments, the trade-off is concrete: trust public benchmarks or invest in local evaluation pipelines that replicate real workloads. This is not only about accuracy but about sovereignty of judgment. An organization that runs models locally for privacy or compliance reasons cannot delegate evaluation to an external judge without asking who controls that judgment. The mapping does not offer definitive answers, but it makes a structural problem visible: the more AI participates in constructing tests, performing tasks, and judging responses, the harder it becomes to distinguish independent evidence from reproduced preference. AI-RADAR covers these topics in the /llm-onpremise section, where the comparison between public benchmarks and local evaluations is part of the trade-off analysis.