LLM evaluation is moving beyond single benchmark scores toward RAG query clarification, calibrated judges and skill profiles. New datasets and frameworks reveal large gaps between exact-match metrics and human-aligned outcomes.
A fully automated framework shows that on 5,000+ simulated dialogues, Remediation Success Rate moves from 42.1% under exact-match to 81.3% under human-calibrated LLM judges. The 39.2-point gap exceeds the combined effect of model, teaching strategy, ...
IBM Research releases ClariEdge, a dataset of about 3,900 examples for spotting ambiguous queries in RAG. A lightweight LoRA obtained through fine-tuning reaches 96.6% accuracy and outperforms larger baselines. The signal: specialized local component...
IBM Research has released ClariEdge, a dataset of about 3,900 examples for recognizing ambiguous queries in RAG systems and generating clarification requests. A lightweight LoRA reaches 96.6% classification accuracy and beats prompted baselines from ...
SkillEval introduces an evaluation framework based on skill profiles automatically discovered by LLMs and estimated with cognitive diagnosis models. Tested on 3,811 models and 9,523 items, it produces more calibrated and predictive profiles. Applicat...
Irene Solaiman, Hugging Face's chief policy officer, has warned that model benchmarking methods are complex and still maturing. Embedded evaluators are not a shortcut. For anyone choosing an LLM, evaluation is a governance issue, not just a technical...
