Static benchmarks are proving insufficient to capture real-world agent behavior, from executable Xiangqi games to decision routing audits. Frontier AI labs are also using hard mathematical problems as new model testbeds, while local coding benchmarks highlight environment sensitivity.
At the Heidelberg Laureate Forum, OpenAI’s Navier–Stokes claim ignited debate. AI companies are turning math into a benchmark, while academics report pressure, costs, and verification shifting outside peer control. The split is not only about theorem...
A community benchmark shows local RTX 5090 setups hitting 96% on coding tasks, close to Claude Code Opus 5.5's 100%. The key variable isn't just the model: with the same GPU and quantization, harness choice swings results from 22% to 96%. MTP variant...
An executable benchmark for Chinese chess shows that LLM agents can point to the right move and still lose. Across 119 tactical endgames, 8,568 multi-turn trajectories and 12 frontier LLMs, closed-loop success collapses compared with static evaluatio...
A paired evaluation across 11 decision points for LLM agents finds hosted Jev outperforms open-weight Laya on 9 tasks, but neither beats chance on zero-shot routing. The pipeline audit turns a claimed 23.9% saving into an actual 4.3%.
The Survprompt framework turns structured clinical data into vignettes and asks pre-trained LLMs to estimate survival without fine-tuning. Across multi-institutional cohorts, frontier models reached cMAE competitive with random survival forests in se...
