The hidden cost of refusal: measuring user disengagement in real-world LLMs
RAW uses LLM stochasticity as a natural experiment to measure the causal effect of refusals on user re-engagement without A/B tests. Applied to 15 deployed models in LMSYS-Chat-1M and WildChat, it finds that for about half of the precisely measurable...