One hundred dollars. That’s what it takes to build an LLM server that handles three simultaneous sessions on a 35-billion-parameter model, each with a 32K token context window and generation throughput around 23 tokens per second. The demonstration comes from a Reddit user who paired two NVIDIA P102-100 cards – mining GPUs available on the second-hand market for a song.

The log details the setup: two cards with 10 GB of VRAM each, a claimed aggregate bandwidth of 448 GB/s, mounted on a system with a Xeon W-2135 processor. The engine is llama.cpp, which loads the Qwen 3.6 35B model in its A3B variant (a Mixture-of-Experts architecture with 3.6 billion active parameters) quantized to 4 bits. The framework allocates three inference slots, each with a 32K context window, and handles requests in parallel. During a test with three identical tasks, prompt processing clocks around 480 tokens per second, while generation settles at roughly 23.5 t/s per user, worth a cumulative 70+ t/s. No cloud needed.

This is much more than a hardware recycling trick. It highlights a strategic pivot in the on-premise AI landscape: llama.cpp’s decision to anchor its stack to CUDA 12.x instead of chasing newer versions. Pascal-based GPUs with compute capability 6.1 are no longer supported by CUDA 13.x. Yet the llama.cpp developers keep compatibility with the 12.8 and 12.9 branches, released in the spring of 2026. As the original poster notes, this “keeps alive a huge slice of users,” protecting the investments of those who can’t or won’t chase the latest accelerator generation.

A move that shifts the balance

Sticking with the 12.x branch is far from neutral. From an enterprise adoption standpoint, it signals that the open-source inference stack is mature enough to set the tempo for the driver ecosystem, rather than the other way around. Anyone evaluating an on-premise deployment today can factor fifth- or sixth-hand cards into their TCO calculation, as long as they pack enough VRAM. The math is stark: $100 versus the thousands required for a last-generation GPU with the same amount of memory.

Second-order effects emerge behind the numbers. The glut of mining GPUs, already depressed after the crypto boom, finds a fresh outlet in open model inference. This pushes hardware costs down for small organizations, independent developers, and IT departments operating air-gapped environments. Being able to run a 35B MoE model with decent performance without sending data outside is a paradigm shift: data sovereignty becomes concretely affordable.

Of course, there are losers. NVIDIA, which is pushing new Blackwell architectures precisely to consolidate the enterprise AI segment, sees its most price-sensitive tier eroded. Cloud providers selling inference APIs lose appeal when a local solution costs the equivalent of a few months’ subscription. The friction between a silicon vendor’s agenda and the reality of an open-source community becomes visible: llama.cpp has no interest in alienating its user base, and that base includes thousands of owners of old cards.

What works for a hobbyist works for a CFO

The point isn’t that a hobbyist can tinker with two recycled cards. It’s that the same logic scales. This P102-100 example proves that retired hardware, when orchestrated with the right runtimes and optimized models, can satisfy real workloads: internal company chat, documentation assistants, product prototypes without touching the public internet. Aggressive quantization and MoE architectures act as multipliers, and the energy cost remains manageable: two 250W cards are unlikely to blow an office electricity budget.

Naturally, limitations remain. 23 t/s latency isn’t that of a premium service, and the context window isn’t infinite. But for many use cases – summarization, translation, basic knowledge Q&A – it’s more than sufficient. And the direction is clear: as compression techniques advance, the on-premise inference entry threshold will keep falling.

For those observing the phenomenon through the lens of digital sovereignty, stories like this deliver a crisp lesson: infrastructure control is no longer the exclusive preserve of those spending millions on hardware. Sometimes a hundred dollars, a framework that refuses to drop legacy drivers, and the will to stop delegating your data to a remote IP are just enough.