A viral and controversial graph

A graph produced by METR (Model Evaluation & Threat Research) has become a benchmark in the world of artificial intelligence (AI), particularly for evaluating the capabilities of large language models (LLMs) such as Anthropic's Claude Opus 4.5. This graph suggests that some AI capabilities are improving at an exponential rate, with new releases exceeding predicted trends.

What the METR graph really measures

The METR graph does not measure AI capabilities in a broad sense. It focuses primarily on coding tasks, evaluating difficulty based on the time it takes humans to complete them. The "time horizon" on the y-axis represents the time it takes humans to complete tasks that a model can successfully perform in 50% of cases. A common mistake is to interpret this value as the length of time the model can operate autonomously.

Criticisms and limitations

Not everyone agrees on the effectiveness of human time as a metric for quantifying AI capabilities. Furthermore, the tasks evaluated do not reflect the complexities of real-world work and focus primarily on coding. Despite these limitations, many experts recognize the value of the METR study as one of the most accurate of its kind, providing a concrete measure of AI progress. For those evaluating on-premise deployments, there are trade-offs to consider; AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.

An imperfect but useful tool

Despite its imperfections and misinterpretations, the METR graph remains a useful tool for evaluating AI progress. It represents an attempt to quantify a rapidly evolving area, providing a concrete benchmark in a field often characterized by vague and hyperbolic claims.