The evaluation of Large Language Models has long been anchored to a simple idea: take a benchmark, compute an average score, and compare models on that single number. But this extreme synthesis hides a far more articulated reality. LLMs are not monoliths: they excel at some skills and fail at others, and aggregating everything into a scalar obscures valuable information. A new framework called SkillEval tries to change this paradigm, bringing tools borrowed from psychometrics into LLM evaluation.

SkillEval does not just produce a new metric. It combines two distinct mechanisms. On one hand, it leverages the meta-cognitive capabilities of LLMs to automatically discover interpretable skills from benchmark questions. On the other, it applies cognitive diagnosis models (CDMs) – originally developed in psychometrics to measure latent abilities of students – to estimate each model's specific abilities on each skill. The result is no longer a report card with a single grade, but a multidimensional profile that describes what a model can do and what it cannot.

The experimentation is extensive: 3,811 LLMs evaluated on 9,523 items distributed across five heterogeneous benchmarks (MATH, BBH, GPQA, MuSR, IFEval). According to the authors, SkillEval produces more calibrated, informative, and concise profiles compared to the average-based paradigm, and it more accurately predicts model behavior on unseen items. In other words, it not only describes the past but anticipates the future.

But the most interesting data comes from downstream applications. The first is dynamic LLM routing: instead of using a single model for all queries, SkillEval assigns each request to the most capable but efficient model for that specific skill. The reported result is a reduction in inference cost of over 60%, with an improvement in overall performance. For those managing on-premise infrastructure, where GPUs are a fixed and expensive resource, this is not an academic curiosity: it means serving more traffic with the same hardware or maintaining quality on less powerful machines, with a direct impact on TCO.

The second application is Computerized Adaptive Testing (CAT), borrowed from adaptive tests used in education. With a trained CDM, an unseen model can be evaluated efficiently, reducing the number of items needed to estimate its abilities. For a company that needs to choose among dozens of models – often with limited resources for benchmarking – this approach changes the timelines and costs of evaluation.

Structurally, SkillEval signals a profound transformation: the benchmark ceases to be a leaderboard and becomes a diagnostic tool. This shifts incentives for model developers. It is no longer enough to chase the highest average on a leaderboard; it becomes more sensible to optimize specific skills, knowing that a specialized model can be routed where needed. Smaller model providers may gain ground because quality on a single skill becomes a marketable asset. Conversely, giant models lose part of their "jack-of-all-trades" advantage if a router can replace them with cheaper models on easy queries.

There is also a signal for evaluation datasets: SkillEval shows that benchmarks share skills and that instruction-tuned models differ from their base versions in ways that are not always obvious. This could lead to a rationalization of test suites, reducing redundancy and focusing efforts on what truly discriminates capabilities.

In an ecosystem rushing toward local deployment and data sovereignty, tools like SkillEval shift the debate from brute force to intelligent resource management. It is no longer just a matter of how many parameters a model has, but how its skills are mapped and exploited.