Imagine building an LLM-based agent to automate business processes. Should you bet on the largest model with the highest token generation speed, perhaps with careful fine-tuning and a hand-crafted system prompt? One user’s on-premise comparison suggests the opposite: for agentic tasks, a smaller, less optimized model can outperform a larger one if it gets the job done with fewer tool calls.

The test, described in a Reddit post, compared a Qwen3.6-27B quantized in INT8 (W8A16) served with vLLM Tensor Parallelism on two Ampere GPUs (decode at 37.7 tokens/s, up to 72 with speculative decoding), against a Nemotron Puzzle-75B-A9B in NVFP4 across three GPUs using vLLM pipeline parallel, capable of 65 tokens/s. Both operated with 131K context windows and fp8 KV caches, without heavy system tuning.

The 27B, left entirely untuned with a neutral system prompt, completed all agentic tasks in 6-9 tool calls, taking between 134 and 190 seconds per task. The 75B, without a hand-tuned profile, was a coin flip; even after tuning it required 13-23 calls and took 221 to 384 seconds. In practice, the model that wastes fewer turns wins—even at half the decode speed.

The report also uncovers another obstacle for anyone benchmarking agents locally: prefix caching. Byte-identical retries caused the XML parser to choke on mangled output, leading to six consecutive failures. Only after cache churn (or adding a per-run nonce to the system prompt) did the test become reproducible. A detail that anyone evaluating agent performance on self-hosted stacks must heed: serving optimizations like prefix caching can make consecutive attempts far from independent, skewing reliability metrics.

Beyond the anecdote, this points to a structural shift in how we think about on-premise LLM infrastructure. While the race for ever-larger models often saturates VRAM and forces distribution across multiple GPUs (with pipeline parallelism, tensor parallelism, and aggressive NVFP4 or INT8 quantization), for agentic workloads—where the model chains multiple reasoning and action steps—end-to-end latency isn’t dominated by single-token generation speed, but by the ability to produce a useful response in few iterations.

This flips hardware provisioning priorities. The fastest card for batch inference no longer cuts it alone; stability of the serving architecture, predictability of response times (which prefix caching can undermine if mishandled), and the ability to run midsize models on fewer GPUs all matter. The Qwen3.6-27B in INT8, for instance, ran on just two GPUs, while the 75B required three with pipeline parallel—a non-trivial resource saving for teams with limited hardware budgets.

Even more intriguing: INT8 quantization on Ampere architecture (fp8 KV, vLLM with speculative decoding) didn’t penalize agent quality; instead, high prefill speeds (764 tokens/s at 76K tokens) combined with fewer turns yielded far lower wall time. This opens space for models trained for logical efficiency rather than sheer size, and for software stacks that prioritize reproducibility and low system latency over raw decode velocity.

Anyone designing agentic pipelines on local hardware would do well to consider that the key parameter is no longer tokens-per-second benchmarks on single prompts, but real time-to-completion on complex tasks, including tool-call overheads and cache variables. And that optimization must not sacrifice test reproducibility—otherwise you risk chasing ghosts.