Topic / Trend Rising

LLM Evaluation and Reliability Gaps

New benchmarks and metrics show that correct answers or token counts do not guarantee sound reasoning or user trust. Nuclear, crisis-support and re-engagement studies expose reliability gaps beyond standard leaderboards.

Detected: 2026-11-22 ยท Updated: 2026-11-22

Related Coverage

2026-11-20 โ€ข ArXiv cs.CL

LLM Complexity: why counting tokens is not a true difficulty measure

A new research study challenges the idea that the number of tokens generated by an LLM indicates problem complexity. The proposed metric, LLM Complexity, looks at the area bounded above by the accuracy-token curve and yields stable difficulty ranking...

#Hardware #LLM On-Premise #Fine-Tuning
2026-11-18 โ€ข ArXiv cs.CL

The hidden cost of refusal: measuring user disengagement in real-world LLMs

RAW uses LLM stochasticity as a natural experiment to measure the causal effect of refusals on user re-engagement without A/B tests. Applied to 15 deployed models in LMSYS-Chat-1M and WildChat, it finds that for about half of the precisely measurable...

#Hardware #LLM On-Premise #Fine-Tuning
2026-11-18 โ€ข ArXiv cs.CL

CrisisBench evaluates crisis counseling AI without exposing real transcripts

A crisis counseling evaluation suite uses de-identified conversations and synthetic patient models. Tested models underperform human counselors on excessive empathy and questioning in single-turn tasks, while retrieval-augmented generation scores hig...

#Hardware #LLM On-Premise #Fine-Tuning
โ† Back to All Topics