Extended reasoning models pay a subtle cost: they generate long chain-of-thought traces, and length is not neutral when serving an LLM on owned hardware. ERR+ starts from an empirical observation across multiple model families: correct traces show more frequent and larger token-level entropy drops during the thinking phase than incorrect ones. The research team turns this into a training signal.

The first phase introduces the Entropy Relief Reward, a bonus proportional to cumulative token-level entropy drops during thinking, log-normalized by response length. Instead of suppressing entropy, the reward targets the resolution of uncertainty and leaves high-entropy exploratory states unconstrained. The second phase adds the Robust Relative Efficiency Reward, which scores each response's length against co-generated peers using a tanh-transformed within-group z-score. The formal analysis indicates that jointly optimizing both objectives creates gradient conflict in early training, which motivates the sequential design. On five datasets, the authors report consistent improvements in both accuracy and response conciseness across model backbones.

The structural signal here goes beyond a single result. RLVR has largely optimized correctness at the end of a trace; ERR+ tries to optimize the internal reasoning process itself. For self-hosted deployments this matters because shorter traces, at equal accuracy, reduce the number of tokens that need to be generated and therefore GPU time, energy consumption, and perceived latency. When hardware is provisioned for predictable workloads, that saving is felt in TCO rather than as a variable cloud bill. It also changes fine-tuning incentives for local and open-weight models, where hardware constraints make token efficiency a competitive parameter alongside VRAM and throughput.

For those evaluating on-premise deployments, AI-RADAR offers analytical frameworks at /llm-onpremise to read these trade-offs. The sequential design is not an implementation detail: the gradient conflict between entropy relief and length efficiency means the two rewards cannot simply be summed early on. The use of a peer-relative length score also avoids imposing an absolute target length across heterogeneous prompts, which is useful when moving from research benchmarks to real deployment conditions.

The code is available on GitHub, but the more important point is that reasoning structure is becoming a trainable optimization target rather than a black box judged only by final answers.