Topic / Trend Stable

LLM Benchmarking and Behavioral Evaluation

Benchmarks are shifting from static knowledge tests toward action, interaction, professional tasks, and behavioral consistency. Studies show major gaps in end-to-end business intelligence, fragile self-descriptions, and systematic personality distortions.

Detected: 2026-09-23 · Updated: 2026-09-23

Related Coverage

2026-09-23 • ArXiv cs.LG

Chat template switches LLM self-referential voice: self-descriptions aren't literal facts

A study on eight open-source instruct models up to 9B parameters shows that the chat template acts as a switch: when present it raises disclaimers like "I'm just an AI" and lowers experiential expressions, and vice versa. Inside three models' activat...

#LLM On-Premise #Fine-Tuning #DevOps

2026-09-22 • ArXiv cs.CL

LLMs and psychology: the benchmark separating recognition, simulation, and refusal

PsyAgentBench reruns five classic psychological experiments on open-weight LLMs under named, blind, canonical, and counterfactual conditions. The results show that effects like Asch conformity depend on the prompt and safety filter, not on a stable b...

#LLM On-Premise #DevOps

2026-09-21 • ArXiv cs.LG

BI-Bench and BI-Agent: the real gap is in data orchestration

End-to-end BI exposes a structural limit of LLMs: below 50 percent accuracy without human guidance. The real gains come from architectures that orchestrate tools and post-training on proprietary data. For on-premise and self-hosted deployments, TCO s...

2026-09-21 • ArXiv cs.LG

BI-Bench and BI-Agent: The LLM Gap in End-to-End Business Intelligence

The BI-Bench benchmark, built from real projects and public dashboards, shows that even frontier LLMs remain below 50% accuracy in end-to-end business intelligence. BI-Agent, a tool-augmented agent that breaks down search, join, and transformation an...

#Hardware #LLM On-Premise #Fine-Tuning

2026-09-18 • ArXiv cs.CL

Subliminal Prompting Beyond Static Geometry: Causal Depth and Multi-Token Confounds

A study on Llama-3.1 and Qwen separates four properties of a hidden prompting channel: fixed geometry, observational readability, causal control, and multi-token measurement. Fixed output-vector similarity loses predictive power from 8B to 70B, while...

#LLM On-Premise

2026-09-18 • ArXiv cs.AI

What Do We Expect from LLMs? Mapping Benchmark Design

A systematic mapping of 14,767 arXiv preprints published between January 2022 and August 2026 shows how LLM benchmarks are shifting toward action, interaction, and professional applications. Model participation is asymmetric: LLM-based scoring grows,...

#LLM On-Premise #DevOps

2026-09-17 • ArXiv cs.CL

When LLMs fake good or bad: response distortion across Dark Triad traits

A study of seven LLMs shows that motivational context systematically shifts the expression of Dark Triad traits. Under fake-good conditions scores fall, under fake-bad they rise, with stronger effects for Machiavellianism and narcissism and in employ...

#Hardware #LLM On-Premise #DevOps

2026-09-17 • ArXiv cs.LG

Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Bounds

A two-tier protocol certifies that an updated model is no worse than the previous one without labeling all traffic. The risk difference concentrates on inputs where the models disagree: only disagreement is checked. Across 14,000+ streams and 785 pai...

#LLM On-Premise #Fine-Tuning #DevOps

← Back to All Topics