Static benchmarks are being challenged by executable tasks, multi-turn trajectories, and paired audits that expose gaps between scoring and real-world performance. The discussion is shifting toward more realistic evaluation of reasoning, decision models, and mathematical claims.
Four open decision models tested on a rented RTX 4090: Laya, d1, Clef-Flash and Lev read 9,534 words and flagged centipede names. Laya is fastest at 3.9 ms per word on llama.cpp; Lev is most accurate on the target but about 13x slower. The test shows...
At the Heidelberg Laureate Forum, OpenAI’s Navier–Stokes claim ignited debate. AI companies are turning math into a benchmark, while academics report pressure, costs, and verification shifting outside peer control. The split is not only about theorem...
An executable benchmark for Chinese chess shows that LLM agents can point to the right move and still lose. Across 119 tactical endgames, 8,568 multi-turn trajectories and 12 frontier LLMs, closed-loop success collapses compared with static evaluatio...
A paired evaluation across 11 decision points for LLM agents finds hosted Jev outperforms open-weight Laya on 9 tasks, but neither beats chance on zero-shot routing. The pipeline audit turns a claimed 23.9% saving into an actual 4.3%.
