A flight simulator in a single HTML file, generated by a language model. Not a conference demo, but the result of a crafty prompt shared on Reddit. The author asked Qwen3.6 35B-A3B to produce a procedural 3D environment with mountains, clouds, and terrain. Nothing revolutionary, except for the technical detail that makes all the difference: the code was generated with the model in 8-bit quantization running on CPU, after the 4-bit GPU version had disappointed. The quality improvement was so stark that the user called the model “phenomenal for its category.” The episode, apparently anecdotal, condenses three crucial junctures for those managing on-premise LLM deployments: the relationship between quantization and output fidelity, the renewed competitiveness of CPUs for inference, and the impact of Mixture of Experts (MoE) architectures on hardware requirements.

The hidden cost of aggressive quantization

Quantization is one of the most widely used levers to fit large models into consumer or enterprise hardware. Reducing weights from 16 to 8 or 4 bits halves (or more) VRAM occupancy, accelerates inference, and enables execution on GPUs with limited memory. However, the loss of precision is not painless. Perplexity benchmarks tend to underestimate the impact on tasks requiring precise adherence to complex instructions, structured code generation, or multi-step reasoning. In the case of the HTML simulator, moving from Q4_K_M (4-bit with some layers at 6-bit) to Q8_0 (uniform 8-bit) gave the model back the ability to faithfully follow a prompt that included a planning phase followed by an implementation phase. The 4-bit quantized version, in all likelihood, introduced enough noise to degrade internal coherence during the generation of a long and articulated output, causing the code structure to derail. While not an absolute novelty, seeing it exemplified on such a complex creative task provides a concrete measure of the phenomenon and reminds us that quantization is not a binary property but a design parameter with direct effects on the perceived quality of the output.
For organizations, this means that the choice of quantization format cannot be delegated solely to speed or memory footprint metrics. A model that produces unreliable code or text requiring continuous corrections nullifies the infrastructure savings obtained with 4-bit. The trade-off must therefore be made explicit and evaluated according to the workload: for classification or summarization tasks, 4-bit may suffice; for structured generation, technical documentation, or code assistance, the higher precision of 8-bit – or even 16-bit when resources permit – can reduce post-processing costs and increase trust in the output. The craft test on Qwen3.6 suggests that already the jump to 8-bit can make the difference between a “average” model and a “surprising” one, even when executed on a processor instead of a graphics card.

MoE architecture shifts the balance toward the CPU

Qwen3.6 35B-A3B is a model with a Mixture of Experts architecture that activates only a fraction of the total parameters for each token. The “A3B” label likely indicates 3 billion active parameters out of a total of 35, a configuration that drastically reduces computational load compared to a dense 35-billion-parameter model. On GPU, this characteristic allows more efficient VRAM usage but does not eliminate the main bottleneck: memory capacity. On CPU, however, access to hundreds of gigabytes of system RAM enables loading the entire model without resorting to offloading techniques or complex partitioning. With a Q8_0 quantized model weighing around 35 GB, a machine with 64 or 128 GB of RAM can host the model without fragmentation and with a memory bandwidth that, while lower than VRAM, is ample for many batch or interactive workflows with low concurrency. The MoE architecture does the rest: only 3 billion parameters are actually traversed per token, keeping the computational cost within manageable limits even for a modern processor with many cores and support for vector instructions.
This scenario overturns a deep-seated perception: the CPU has always been considered a fallback for LLM inference, a solution viable only when GPUs are lacking. Yet with sparse-structure models and careful quantization, the CPU can become a first-choice platform for workloads where output quality takes precedence over latency. This does not mean imagining a generalized takeover, but recognizing that there is a class of applications – code generation, reporting, data analysis in asynchronous mode – where a CPU server with plenty of RAM can offer a quality-price ratio that is hard to match with GPUs, especially when considering the total cost of ownership including energy consumption, cooling, and hardware refresh cycles. Moreover, the ability to scale horizontally with multiple CPU nodes, perhaps leveraging machines already present in the server room, opens interesting prospects for distributed inference without having to upheave existing infrastructure.

Total cost of ownership: RAM as a strategic asset

For an organization evaluating an on-premise LLM deployment, the TCO calculation often revolves around the price of GPUs. A card with 24 GB of VRAM can cost several thousand euros, and for models with 35 billion parameters, at least two cards or a configuration with partial offloading would be needed, with the associated bandwidth and latency complications. A server machine with a high core-density CPU (e.g., 32 or 64 cores) and 128 GB of DDR5 RAM has a comparable absolute cost, but offers a more flexible environment: RAM can be expanded up to terabytes at marginal cost, whereas VRAM on GPUs is a physical constraint that is difficult to overcome except by buying more expensive cards or adopting multi-GPU architectures that inflate the system budget. The Qwen3.6 8-bit test shows that on a RAM configuration considered standard for servers today, it is possible to run a model with high output quality without compromising on weight precision.
The equation changes further when considering the hardware lifecycle. GPUs, driven by frantic innovation and supply bottlenecks, follow a rapid obsolescence curve, with new generations succeeding each other every 18-24 months. CPU servers based on mature platforms have a longer cycle and benefit from already consolidated economies of scale. Even energy consumption, in a CPU inference scenario, can be easier to manage within data centers designed for general-purpose workloads, without requiring specialized cooling systems. For organizations that already have traditional servers, the prospect of repurposing part of the unused computational capacity to serve on-premise LLMs, simply by adding RAM and using 8-bit quantized models, could turn a cost center into an enabling asset for generative AI behind the firewall.
Nonetheless, the latency limitation must not be overlooked. On CPU, the time to generate a token is higher than on a modern GPU, and for interactive applications with many concurrent users, the slowness can become unacceptable. However, for overnight batch workloads, documentation generation pipelines, or assistants operating in asynchronous mode, latency is not the critical factor. The key variable becomes memory bandwidth, which on server platforms with eight or more DDR5 channels can reach values sufficient to sustain an acceptable token throughput for many use cases. Monitoring the evolution of CPUs with integrated AI accelerators (like NPUs or vector engines) thus becomes a signal not to be underestimated: if such units become common on server processors as well, the gap with GPUs could narrow further for inference workloads.

Data sovereignty and output reliability: two sides of the same coin

The principle of data sovereignty pushes many organizations to prefer self-hosted solutions for generative AI, avoiding sending sensitive information to external cloud services. But sovereignty is not just a matter of the physical location of data: if the on-premise model produces unreliable output, the cost of corrections can wipe out the benefits of direct control. The flight simulator example shows how a seemingly subtle difference in quantization format can make the model useful or unusable for a complex task. If the goal is to generate code to integrate into a software product, low fidelity creates technical debt and requires manual intervention, nullifying automation. The choice of quantization format thus becomes an integral part of the sovereignty strategy: it is not enough to keep data in-house; one needs a model that respects instructions without introducing errors that are hard to detect.
This awareness prompts a rethinking of the entire deployment stack. Instead of focusing solely on speed with GPUs and extreme quantizations, some teams might adopt a hybrid approach: multiple CPU instances for workloads where quality is paramount, and GPUs for real-time services that tolerate reduced precision. Queue management and load balancing would become more complex, but they would allow the best of both architectures to be exploited. In the specific case of Qwen3.6, a model that proves capable of producing complex code at 8-bit on CPU could become the engine of an internal development tool, where developers submit a request, wait a few extra seconds, and get a directly usable result. The cloud alternative would have lower latencies, but it would entail sending code and specifications outside the corporate perimeter – a risk that many regulated sectors cannot afford.
Trust in the output is the true enabler of sovereignty. If the model stumbles in subtle ways, data control serves little purpose because human decision-making must still verify every single response, negating productivity gains. A conservatively quantized CPU model, possibly with a large context window and good instruction adherence, can instead be delegated to generation tasks with a reduced level of supervision. The craft test of Qwen3.6 thus becomes a wake-up call: the pursuit of speed and aggressive compression must not obscure the reason for adopting an LLM, namely the ability to produce meaningful and reliable outputs.

What to watch in the coming months

The episode signals some trajectories that organizations would do well to monitor. The first is the evolution of MoE models. If vendors continue to release architectures with a favorable ratio of total to active parameters, CPU inference will become progressively more feasible even for nominally large models. The arrival of inference frameworks optimized for CPU, such as back-ends that leverage AVX-512 or AMX instructions on Intel processors and Apple's AMX/Metal units, could further accelerate this trend. The second signal is the direction of quantization formats. If research produces compression techniques that maintain 8-bit quality with a 4-bit footprint, the boundary between CPU and GPU will shift again. But for now, pragmatism suggests testing models under real-world usage conditions, evaluating perceived quality and not just abstract benchmarks, before deciding on the deployment format.
Finally, the server hardware market could respond with hybrid platforms that integrate neural accelerators alongside CPUs with many memory channels, explicitly designed for LLM inference. If such solutions reach affordable price points, today's trade-off between fidelity and speed could thin out, opening a third path for self-hosting: systems that offer the memory bandwidth of a GPU with the expandability of RAM. In the meantime, the home test of a flight simulator reminds us that every architectural choice, from quantization to hardware, has a direct impact on what the end user sees and uses, and that well-designed on-premise is not a surrender of quality, but the condition for being able to control it all the way.