The line between an LLM that runs with dignity on a Raspberry Pi and one that collapses onto itself is almost never a straight line. It’s a cliff, and the charts that usually circulate hide it. Making this concrete is an independent analysis posted on Reddit by user crusaderky, who put LiquidAI’s LFM2.5-2.6B model through its paces — a tiny gem with only 2.6 billion parameters that, according to official benchmarks, holds its own against much larger architectures.

The experiment is surgical: for every possible combination of GGUF model quantization and KV cache quantization, perplexity was calculated on test sets, looking for the best quality-memory trade-off on Raspberry Pi with 8 and 4 GB of RAM. The results tear open more than one blind spot about how easy it is to be deceived by quantization metrics when running on-device inference.

Let’s start with the facts. On an 8 GB Raspberry Pi, LFM2.5-2.6B runs with no material degradation — a datum that alone rewrites expectations for local language model deployment. On the 4 GB variant, degradation exists but is “contained.” The real surprise, however, comes from digging into the quantization details: the Q4_K_M format, often used as a safe choice to save VRAM, must be categorically avoided with this model. Its quality curve does not descend smoothly; at some point it plummets, and it does so without any alarm from traditional metrics — such as logarithmic KLD or Top-1% token accuracy. In fact, those numbers tell a story of linear, predictable degradation, whereas reality is a sudden jump that can silently invalidate the model’s responses.

There is another lesson, delicate for anyone working on model alignment: abliteration, the technique for removing unwanted behaviors, carries a flat cost of about 0.075 KLD, a fixed price that does not depend on the chosen configuration. This means every safety intervention on the LLM shaves off a piece of predictive capability, and this toll must be accounted for when deciding whether and how to deploy the model in production.

Illusion of linearity
The real twist of the report is exactly the cliff-like behavior of quality. In many charts, degradation appears gradual only because common metrics operate in compressed (logarithmic) space or focus on distribution tails. But when looking at real perplexity, the drop is sharp. It’s a conceptual bug report: anyone assessing whether a quantized model can work for a specific task — on-prem customer service, document summarization in air-gapped environments, chatbots for public administration — risks a blunder if they blindly trust aggregate measures without testing actual behavior on the target hardware.

What does all this mean for the on-premise AI landscape? First, that the road of small, “pocket-sized” models is more mature than many think. LFM2.5-2.6B shows that you don’t need data center GPUs to have a useful LLM: devices costing less than a hundred euros suffice. The question is no longer whether it’s possible, but with what precautions. For those evaluating on-site deployment for data sovereignty reasons or to contain TCO, crusaderky’s analysis offers a compass: always test real quality on your own machine, don’t trust standard metrics, and be wary of quants that promise too much memory savings without showing the bill.

This is not a technical detail for the few. It’s the kind of information that shifts the frontier between what can be put into local production and what remains confined to the cloud. And it reminds everyone of an uncomfortable truth: in the world of quantization, the straight line is almost always an optical illusion.