Fine-tuning on popular coding benchmarks does not transfer to broader development suites, and adversarial fuzzing or specialized domains expose fragile generalization in LLMs and MLLMs. The community is shifting toward execution-based and action-oriented evaluation to avoid benchmark overfitting.
Fine-tuning on SWE-bench does not transfer capability to suites like Django or LiveCodeBench. The benchmark becomes a training target and stops measuring general ability. For self-hosted LLM operators, an inflated score distorts VRAM, quantization, a...
Optimizing an LLM on popular coding benchmarks does not improve general programming ability. A new Django-based suite shows rankings often fail to generalize, and fine-tuning on SWE-bench yields limited or no gains on other tasks. Evaluation must be ...
A controlled six-condition ablation challenges a common assumption: in LLM self-reflection for armed conflict forecasting, structured diagnostic questions and uncertainty taxonomies add no measurable value. Typed action routing drives consistent gain...
Backtrader-Bench introduces two pipelines to evaluate LLM agents in algorithmic trading: one generates multiple-choice questions from backtest configurations with independent verification, the other mines harder questions that require code execution....
A dedicated benchmark tests 33 large language models on solid waste management tasks. The best model hits nearly 95% accuracy on easy questions, but on hard ones the average drops to 42.5%. Calculation, experimental design, and urban planning remain ...
A new evaluation framework pairs a unified taxonomy benchmark with self-adaptive multimodal fuzzing (SAMF) and shows that state-of-the-art MLLMs degrade under stress, revealing a gap between reasoning and factual grounding. RL alignment even worsens ...
