The leap is hard to ignore. inclusionAI has published Ling-3.1-flash on Hugging Face, a Large Language Model with 560 billion total parameters, 25 billion active parameters per token, and a 1-million-token context window. The discussion starts from an immediate comparison: the previous Ling 3.0 was listed at 127 billion parameters. Reading these numbers as a simple scale increase would be reductive.

The gap between 560B total and 25B active per token is the detail that explains the model's nature. This points to a mixture-of-experts architecture: for each token, only part of the parameters is used. This reduces the amount of computation per token compared to a dense model of the same overall size. The computational cost per token approaches that of a 25-billion-parameter model, not a 560-billion one. But the memory cost remains tied to the total parameters.

Where the cost hides

For self-hosted deployments, the problem is not the number of active parameters, but the amount of VRAM required to keep all experts in memory. In FP16, 560 billion parameters occupy roughly 1.12 TB for weights alone, before considering the KV cache. With a 1-million-token context window, the cache grows significantly and can become the most expensive memory line in scenarios with many concurrent requests or long sequences.

Quantization therefore becomes the first tool to reduce the requirement: moving to INT8 or 4-bit cuts the weight footprint substantially, but introduces a trade-off in quality and serving complexity. Even with aggressive compression, the model remains out of reach for a single consumer GPU. It requires machines with high VRAM, distribution across multiple cards, and expert offloading strategies, with direct effects on latency, throughput, and TCO.

This changes incentives for on-premise model operators. The race to 1-million-token context windows is not neutral: it shifts the bottleneck from compute power to memory capacity and KV cache management. Infrastructure sized only on active parameters risks hitting memory saturation long before saturating compute cores. It is a structural signal: models with extreme windows force local operations to invest in cache paging, KV quantization, and memory distribution, not just faster accelerators.

For those evaluating on-premise deployment, there are trade-offs between memory cost, response latency, and model quality. The Ling-3.1-flash spec sheet, however partial, works as a reminder: active parameters tell only half the story. The other half lies in the memory needed to serve all experts and manage very long contexts without exploding operational cost. AI-RADAR offers analytical frameworks at /llm-onpremise for comparing these variables without reducing the choice to a single benchmark.