Research shows that optimizing LLMs on coding benchmarks does not transfer to general programming capability, and hidden reasoning extraction reveals benchmark contamination and overthinking. New evaluation approaches emphasize real code execution and calibration rather than ranking scores.
A controlled clinical benchmark on gpt-4.1-nano shows 93.5% accuracy but imperfect calibration: confidence rises with evidence distance from the diagnostic boundary and falls with missing information, yet remains too high in moderate, conflicting err...
Fine-tuning on SWE-bench does not transfer capability to suites like Django or LiveCodeBench. The benchmark becomes a training target and stops measuring general ability. For self-hosted LLM operators, an inflated score distorts VRAM, quantization, a...
Optimizing an LLM on popular coding benchmarks does not improve general programming ability. A new Django-based suite shows rankings often fail to generalize, and fine-tuning on SWE-bench yields limited or no gains on other tasks. Evaluation must be ...
Backtrader-Bench introduces two pipelines to evaluate LLM agents in algorithmic trading: one generates multiple-choice questions from backtest configurations with independent verification, the other mines harder questions that require code execution....
A paper shows how to extract all reasoning tokens from Claude and GPT models. It reveals widespread overthinking, benchmarks tainted by memorization, and China’s exploitation of the gap to distill frontier models. Closing this leak redefines the real...
