Refusal in large language models has mostly been treated as a safety matter. But when a production model says no, it is not only applying a policy: it is shaping the behavior of the person on the other side. Deployed models handle hundreds of millions of conversations a day and refuse a non-trivial share of requests. Yet until now no one had measured on a commercial model how much that refusal costs in terms of users who stop engaging or reduce their involvement. It is a blind spot for the consumer-facing platforms built on these models, and an operational problem for those integrating them via fixed APIs without being able to touch the system's behavior.

RAW arises from this constraint. A/B testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms using fixed APIs. The method instead uses LLM stochasticity as a natural experiment: for near-identical prompts in existing logs, the same model sometimes refuses and sometimes does not. That variation, although not explicitly randomized, makes it possible to estimate the causal effect of refusal on user re-engagement. Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, the researchers found that for about half of the precisely measurable models, a refusal costs between 3 and 17 percentage points of subsequent re-engagement. The rest show no distinguishable effect. In one deployment, the positive point estimate was traced back to persistent users retrying the refused request.

The finding is not just academic. It shifts refusal from the safety perimeter to the economics of the product: a model that refuses too rigidly can erode retention without the provider noticing, because aggregate usage numbers do not distinguish a user who leaves after a no from one who returns. The ability to measure this effect on existing logs, at about $0.30 per 1,000 conversations, changes incentives for self-hosted deployers too. Those who run models in-house often have direct access to logs but not the scale to run controlled experiments on end users. RAW offers a way to interrogate data they already own, without modifying the model or exposing conversations to third parties. In contexts where data sovereignty is a constraint, this is a concrete advantage: measurement happens on local logs and requires neither access to the weights nor sharing traffic with external services.

There is also a structural reading. Platforms consuming models through fixed APIs cannot change the model's behavior, but they can measure the impact of refusals on their own users and decide whether to intervene at the orchestration, prompt, or model-selection level. This partially redistributes control: the model provider still defines refusal policies, but the deployer gains a tool to assess their economic cost on its own traffic. That is not a minor detail in a market where TCO is often calculated only on GPUs, latency, and throughput, leaving out the value lost to disengagement.

The researchers released labels, moderation scores, prompt embeddings, and pipeline code. That is the kind of contribution that makes measurement replicable even for teams without a frontier lab.