halogen 0.12.0 started from a specific report: the server was degrading as context grew deeper. The updated numbers, published by peonist-ai on a Ryzen AI Max+ 395 system with 128 GB of unified memory, show the fix. At 1,004,581 tokens of context, with greedy decoding and 64 generated tokens, Qwen3.8-Flash-Next decode moves from 27.3 to 38.3 tokens per second; cold prefill drops from 21.2 to 17.9 minutes. At 258,794 tokens, the gains are smaller: decode goes from 42.9 to 45.0 tok/s and prefill from 1,086 to 1,114 tok/s. The 32k ten-prompt served mean remains unchanged.

The most revealing part is where the improvement lands. The largest percentage change appears at extreme context, not in short-benchmark territory. That suggests a targeted fix for long-context state management rather than a generic speedup. The server runs via podman with HALOGEN_CTX=1048576 and HALOGEN_ROPE_YARN=4, a configuration that requires the 128 GB machine. On a million-token context, memory footprint and caching strategy determine whether the system is usable at all.

The first-token cost deserves close attention. An 18-minute cold prefill can look prohibitive, but it must be read alongside the prompt-cache path: a follow-up turn on the same context reaches the first token in about 0.55 seconds. That changes the realistic applications. If the context is reloaded every time, 18 minutes remains a barrier; if it stays resident for repeated questions over long documents, the system becomes interactive. The architectural choice is therefore tied to usage pattern: one-time ingestion plus multiple queries is very different from single-shot batch.

From an on-premise perspective, this result signals that local long context does not necessarily require a data center. Strix Halo's unified memory removes CPU-GPU transfers that typically become a bottleneck on discrete-GPU systems as context grows. The constraint shifts from raw compute to memory capacity and serving-runtime quality. This has TCO implications: for such workloads, 128 GB of unified memory on a single node may replace more expensive and complex setups, but it imposes clear limits on cold prefill and depends heavily on context reuse.

It is not a final destination. The author asks anyone with 1M-token sweeps to rerun them on 0.12.0, underlining how few reproducible benchmarks exist in this range. These numbers come from precise conditions: one cold request for the 262k and 1M rows, cold path, identical prompts, default speculative drafter. For teams evaluating self-hosted deployments, trade-offs around prefill, caching, and unified memory sit at the center of the decision; AI-RADAR explores those trade-offs on /llm-onpremise, with analysis of constraints rather than pre-packaged advice.