Studies show that coding benchmark optimization does not transfer to general ability, medical models remain miscalibrated, and self-reflection or research agents struggle with real judgment. The focus is shifting toward robustness, calibration, and hallucination detection.
LongNovel is a multi-scale, bilingual Chinese-English benchmark for detecting hallucinations in long novel summaries. Built on 29 Chinese novels from 16k to 100k tokens and BookSum data, it identifies eight hallucination types. The test set was manua...
A Princeton-led study put Claude Opus 4.8 agents to work on unpublished NeurIPS 2026 research questions. The agents handled engineering tasks but lacked the judgment and creativity needed for open-ended research, and both papers were rejected. The fi...
A controlled clinical benchmark on gpt-4.1-nano shows 93.5% accuracy but imperfect calibration: confidence rises with evidence distance from the diagnostic boundary and falls with missing information, yet remains too high in moderate, conflicting err...
Fine-tuning on SWE-bench does not transfer capability to suites like Django or LiveCodeBench. The benchmark becomes a training target and stops measuring general ability. For self-hosted LLM operators, an inflated score distorts VRAM, quantization, a...
Optimizing an LLM on popular coding benchmarks does not improve general programming ability. A new Django-based suite shows rankings often fail to generalize, and fine-tuning on SWE-bench yields limited or no gains on other tasks. Evaluation must be ...
A controlled six-condition ablation challenges a common assumption: in LLM self-reflection for armed conflict forecasting, structured diagnostic questions and uncertainty taxonomies add no measurable value. Typed action routing drives consistent gain...
