Two RTX 3090s linked over PCIe Gen4 without NVLink, a Qwen3.8-27B model with AutoRound INT4 quantization, and a modified vLLM: the combination produced 218 tokens per second on a single request. The result, shared on Reddit and measured with the Club-3090 benchmark suite, is not just a lab number. It is a snapshot of the concrete trade-offs that now govern self-hosted inference for mid-size models.

The setup is bare-metal: two RTX 3090 cards peaking at 22.3 GB of VRAM each, power capped at 220 and 250 W, PCIe Gen4 x16 on both slots, no NVLink but patched P2P. On top runs vLLM 0.26.1rc1 with AutoRound INT4 quantization (group 128) and a DFlash2 draft model for speculative decoding. The maximum context reaches 131k tokens; the drafter alone consumes about 13.5 GB of VRAM.

Focusing only on the 218 tok/s would miss the point. The more telling number is the cost of the drafter. To accelerate generation, the system must host two models instead of one: the quantized main model and the draft model that proposes tokens. The price is a significant chunk of memory and a lower effective context ceiling. Without DFlash2, the context window would be larger, but decode speed would likely collapse. That is the classic trade-off of speculative decoding: spend extra VRAM to propose multiple tokens in parallel, hoping the main model validates enough of them. In this case, out of 7 proposed tokens, an average of 3.35 are accepted, for an acceptance rate of 47.8%. It is not a perfect yield, but it is enough to push real throughput above 200 tok/s.

The full measurements separate a narrative scenario from a code scenario. In narrative, decode runs at 120.1 tok/s; in code, 218.3. Wall throughput drops slightly to 117.7 and 204.8 tok/s. Time to first token remains contained at 168 ms for narrative and 178 ms for code. These are interactive numbers, not batch-serving figures: the benchmark measures a single request, not concurrent load. For someone looking to replace a cloud endpoint for conversations or code assistance, this is a solid base. For teams that need to serve many users in parallel, the picture is different.

This performance did not happen by accident. The author describes putting the setup together quickly and using Kimi K3 for vLLM fixes, including custom changes that make the system boot cleanly. That is the detail that changes incentives for anyone evaluating an on-premise deployment. The GPU savings are real, but TCO does not stop at hardware purchase: it includes serving skills, quantization work, runtime debugging, and likely ongoing maintenance. For a team that already has those capabilities, two RTX 3090s become a very interesting local inference platform. For an organization starting from zero, the hidden integration cost can outweigh the cloud savings.

The result also points to a broader shift. Lowering inference costs for 27-billion-parameter models now depends on two levers: aggressive quantization and speculative decoding. The goal is no longer just to add compute, but to reduce memory footprint and make better use of already generated tokens. On consumer hardware without NVLink, the limit is not peak throughput but the ability to hold main model, drafter, and context together. That shifts the advantage toward teams that can modify code and configuration, not simply buy more GPUs.

For those evaluating on-premise deployments, AI-RADAR offers analytical frameworks at /llm-onpremise to read these trade-offs without reducing the choice to a single benchmark. The author hints that more performance may be available: the setup was assembled quickly with custom fixes, leaving room for further optimization.