Three and a half hours, one hundred turns, two conversations and a log-scale graph: enough to see where a local LLM with extended context starts to break. The experiment runs on a MacBook Pro M5 Max with 128 GB of unified memory, uses llama.cpp b10686 with Metal, 12 threads and batch 2048, and a 78.9 GB Qwen3.8-Flash-Next in 2-bit quantization, pushing the context slot to 358,400 tokens via YaRN from a native window of 262,144 tokens.

The immediate takeaway is that the hardware holds up better than expected. Weights and a full 350K-token KV fit under the 96 GB GPU wired memory limit, with no system hack. The green prompt-processing curve shows two distinct behaviors: full cold prefills drop from 1,561 tokens/s at the first checkpoint to about 318 tokens/s at 111K tokens; normal turns, which benefit from prefix reuse, process only the delta and range from 77 to 854 tokens/s out to 169K tokens. Generation, the red curve, tapers cleanly: roughly 30–35 tokens/s at small context, 21 at 45K, 13–15 between 100K and 125K, and 11.5 at 169K. The only anomaly is a dip to 7.7 tokens/s around 140K, caused by macOS Low Power Mode.

The decode numbers should not be read as base-model speed: they are effective throughput with ngram-mod spec decode enabled, with draft acceptance ranging from 0 to 81 percent depending on content. The longest prefill, 333 seconds for a 105K prompt, happened only once after a roughly twenty-minute idle gap reduced the slot to just the 5.5K system prefix. In other turns, prefix reuse cuts prefill to a few seconds. The maximum slot capacity was never filled: the deepest point reached was 169,425 tokens. The second conversation stopped around 125K tokens when capture was interrupted.

The critical issue, however, is not speed but quality. Beyond 100K tokens the model began mixing up user messages with its own prior output, a role-confusion problem that worsened with use. The author ruled out KV quantization, because the KV was running in fp16, and rope extrapolation, because the worst turns were well below the native 262K window. Two suspects remain: 2-bit model quantization and the long-context quality of a preview model.

This is a structural signal, not just a forum anecdote. Until recently the local inference constraint was memory and bandwidth; this test shows that, on high-end consumer hardware, the bottleneck shifts to the robustness of the compressed model and its ability to keep conversational roles distinct at the end of the context. A 350K-token slot is possible, but its practical value collapses if the model starts responding on behalf of the user.

For anyone evaluating self-hosted deployments, the lesson is clear: comparing gigabytes of memory or claimed tokens/s is not enough. You need to test behavior beyond 100K tokens with aggressive quantization, because that is where defects emerge that no synthetic speed benchmark captures. AI-RADAR provides analytical frameworks in the /llm-onpremise section for evaluating these trade-offs without reducing them to a single metric.

There is also a hardware flip side. The fact that a laptop with unified memory can hold weights and a full 350K-token KV without system hacks makes long-context experimentation far more accessible. But the 2-bit compression needed to fit 79 GB models becomes the prime suspect for the quality defects. That helps explain why many enterprise environments prefer more conservative quantization levels or smaller models with native windows, even at the cost of more hardware: predictability matters more than the maximum declared context.

It remains to be seen whether the observed problem is solvable with a final Qwen3.8-Flash-Next checkpoint in 2-bit quantization or whether the preview model itself is not trained to remain coherent beyond 100K tokens. The answer will not come from the graph, but from the behavior of those who try to use these models for real work.