Studies show coding benchmark optimizations do not transfer to general programming ability, and benchmark-focused fine-tuning distorts rankings. Medical, self-reflection, framing, and AI text-detector evaluations also reveal calibration gaps, uneven moral judgment, and detector limitations.
A controlled clinical benchmark on gpt-4.1-nano shows 93.5% accuracy but imperfect calibration: confidence rises with evidence distance from the diagnostic boundary and falls with missing information, yet remains too high in moderate, conflicting err...
Fine-tuning on SWE-bench does not transfer capability to suites like Django or LiveCodeBench. The benchmark becomes a training target and stops measuring general ability. For self-hosted LLM operators, an inflated score distorts VRAM, quantization, a...
Optimizing an LLM on popular coding benchmarks does not improve general programming ability. A new Django-based suite shows rankings often fail to generalize, and fine-tuning on SWE-bench yields limited or no gains on other tasks. Evaluation must be ...
A controlled six-condition ablation challenges a common assumption: in LLM self-reflection for armed conflict forecasting, structured diagnostic questions and uncertainty taxonomies add no measurable value. Typed action routing drives consistent gain...
A controlled study shows commercial AI text detectors cannot distinguish assisted editing from fully LLM-generated drafts. Light, guideline-compliant edits are flagged in 64–80% of cases, while recent originals only in 9–15%. Honest AI use carries hi...
A study of 28,592 French headlines shows LLMs reliably detect conflict and strategic-game frames but struggle with normative judgments. The findings guide the design of local annotation pipelines that separate robust from fragile constructs, mitigati...
