The starting figure is uncomfortable for anyone automating web tasks with predictive models: in 3,000 predictions generated by ten frozen open models, human evaluators considered 86.1% semantically plausible. Yet when those models must decide whether an action can be executed, requires clarification, or should be blocked, behavior becomes asymmetric. The picture shows low false-safe execution, poor recall of ASK cases, and a strong preference for BLOCK.

Plausible future prediction does not necessarily preserve policy-decisive information. SafeWebWorld benchmark measures safety sufficiency: the ability to distinguish EXECUTED, ASK, and BLOCKED actions across five web-safety domains using action-conditioned futures, policy-decisive evidence, and policy-conditioned decisions. Under a controlled Security-Aligned Supervised Fine-Tuning protocol, evidence preservation improves substantially while authorization-sensitive decisions deteriorate. It is a counterintuitive result: fine-tuning helps the model retain cues but degrades its ability to use them for correct authorization.

The gap between plausibility and authorization

To determine whether the drop comes from loss of authorization-sensitive score structure or from a failure in decision readout, the work introduces Latent Authorisation Recovery (LAR). Frozen-score recalibration shows substantial but backbone-dependent recovery. Paired recovery adds only a modest gain. Within-family and matched-authorization evaluations remain poor. In other words, aggregate recoverability does not imply robust fine-grained authorization behavior.

This has structural consequences for anyone evaluating self-hosted models. A model can look solid because it produces coherent predictions, but its decision boundary can be fragile at specific points that aggregate metrics do not surface. Semantic plausibility is not a sufficient proxy for authorization robustness. The signal for on-premise stacks is clear: evidence preservation must be separated from decision calibration, and the two levels must be tested independently. In air-gapped or regulated settings, where the model operates without continuous human supervision, a biased preference for BLOCK can reduce dangerous errors but creates operational bottlenecks and unnecessary escalation requests; conversely, false-safe execution is rarer in the observed models but more critical when it occurs.

Latent recovery: a partial repair

The fact that recovery depends on the backbone rules out a universal correction. For teams managing local inference pipelines, acceptance tests cannot be limited to language benchmarks or plausibility rates: they must include boundary cases between ASK and BLOCK and scenarios where policy changes. The source does not provide hardware specifications or costs, but the logic applies to any on-premise inference pipeline: authentic safety evaluation must observe policy-conditioned decisions, not just prediction quality. For those evaluating on-premise deployment, AI-RADAR offers analytical frameworks at /llm-onpremise to separate trade-offs between control, latency, and decision robustness.

In the end, the conclusion is sharp: plausible prediction, evidence preservation, and aggregate recovery are each insufficient on their own to establish robust authorization.