It looked like a mystery. On a Mac Studio M3 Ultra with 96 GB of unified memory, Qwen3.5-122B would run smoothly, but after a few messages in an agentic coding session the model would fall into icy silence: 3 to 5 minutes before starting to generate a response. The context, formally “warm”, was actually hiding three flaws in the serving stack that short-circuited the KV cache.

Developer marzukia isolated the bugs in his qMLX fork – a derivative of rapid-mlx optimized for Apple Silicon – and fixed them, open-sourcing the code and a benchmark script that separates prefill and decode metrics. The result is a paradigm shift for anyone using long-context models on local hardware: with 53,000 tokens in cache, only 33 needed recalculation, bringing inference back to sub-second speed.

A technical knot hiding in plain sight

The first culprit was a matter of identity: the unique message ID inside the system prompt broke byte-exact KV cache matching, forcing a full prompt recompute on every turn even when the context was unchanged. The second bug concerned interrupted streaming replies: the partial generation was never persisted, creating history divergence that invalidated later lookups. Finally, a background checkpoint writer produced unmatchable snapshots that crowded out valid ones and triggered aggressive cache eviction.

Once these three conditions were fixed, the impact was dramatic. Prefill – the operation that turns a conversation into a computable state – collapsed from several minutes to a few tens of milliseconds. The model started responding as expected from a 122-billion-parameter LLM running on hardware that, while lacking dedicated VRAM, leverages the high bandwidth of unified memory.

Forking as realism

One detail cuts through everything else: the optimizations introduced are so specific to Qwen’s hybrid attention architecture that the author decided not to propose a merge into the upstream project. That choice signals something deeper than the usual fork divergence. On local hardware, where every watt and every byte of cache count, serving strategies for modern models are becoming less and less generalizable. Hybrid attention, which combines sliding-window and global attention mechanisms, demands caching logic that fits poorly with stacks designed for more uniform architectures.

What it means is that building a genuinely usable coding assistant on an on-premise machine no longer requires just a powerful model and adequate hardware: it takes specialized forks, maintained by small communities or individuals, capable of squeezing every architectural peculiarity. This is not an oddity – it’s the direction local deployment is heading as models grow more complex and diverge from each other. The qMLX case shows that the line between “it works” and “it’s unusable” can be drawn by three incremental caching bugs, invisible without forensic analysis.

Anyone evaluating a workstation like the Mac Studio for agentic workloads shouldn’t only look at the spec sheet – 96 GB of unified memory looks generous, but the devil lives in the serving details. The path to long-context on-premise inference is paved with similar traps. And when the fix comes not from a vendor but from an independent developer who decides to fork, it means that controlling your own stack is the only real lever for reliability.