In a move that redefines the ambitions of local inference, thinkingmachines has released Inkling-Small, a Large Language Model that activates only 12 billion of its 276 billion total parameters, while offering a 1 million token context window. The model is immediately available in two quantized variants — NVFP4 and GGUF — squarely aimed at deployment on owned hardware, far from cloud data centers.

The choice of a Mixture of Experts (MoE) architecture is no accident: keeping 276 billion parameters in memory would be prohibitive, but with just 12 billion active per token, throughput can approach that of much smaller models. Yet Inkling-Small does not stop at balancing capability and compute costs: it pushes the context window to 1 million tokens, a length that until recently was the exclusive domain of closed models hosted on cloud infrastructure like GPT-4 Turbo or Claude 3.

The memory challenge is not (only) in the weights

The real bottleneck when running a million-token context locally is not the parameters — 12 billion at 4-bit consume less than 6 GB of VRAM — but the key-value (KV) cache. Handling 1 million tokens without degrading performance can easily require tens of gigabytes of memory, making GPUs with high capacity, such as NVIDIA H100s or, prospectively, high-end consumer cards with native FP4 support, a necessity.

This is where thinkingmachines' strategy becomes particularly interesting. The NVFP4 format is designed to exploit NVIDIA hardware accelerators for 4-bit floating point, a technology that the new RTX 50-series makes accessible outside traditional data centers. In parallel, the GGUF files prepared by Unsloth provide a more universal route, compatible with llama.cpp and Ollama stacks, tapping into an already mature self-hosted ecosystem.

What changes for on-premise deployment evaluations

Delivering an LLM with a 1 million token context that can run on owned hardware reshapes the landscape for sectors like legal, healthcare, and scientific research, where entire document corpora must be analyzed without handing data control to third parties. The combination of MoE and aggressive quantization makes a Total Cost of Ownership that doesn't explode with prompt length thinkable, shifting the needle on the performance-versus-data-sovereignty trade-off.

Inkling-Small is not the first model to offer quantized versions for local inference, but its arrival — with NVFP4 and GGUF ready to use — signals a maturing market: model makers are beginning to design with the GPU under your desk in mind, not just the multi-million-dollar cluster. The question of whether managing the KV cache over such long contexts will become practical without radical optimizations remains open, but for now, those who need deep analysis without sacrificing physical control over data have a new ally.