In the LocalLLaMA subreddit, the suspicion that the community is measuring the wrong thing has taken the form of a proposal. User Express_Quail_1493 calls for a revolution: fewer threads about token/s speed, more private quality benchmarks, especially when comparing different quantization levels. The rule would be simple: everyone builds their own benchmark, keeps the test dataset private, and shares only the charts with results. The goal is not a new public leaderboard, but a network of comparable signals among users running LLMs locally.

The issue is not the metric itself. Token/s measures inference throughput and makes sense when you need to understand whether a GPU or CPU can sustain a workload. But when it becomes the dominant parameter, it rewards configurations and models optimized to generate output quickly, not to answer well. In a self-hosted setting, where VRAM and TCO matter as much as answer quality, speed is a necessary but not sufficient condition. Yet the subreddit gets flooded every few hours with threads devoted to that single number.

The proposal hits a critical point: quantization. Moving a model from FP16 to INT8 or more aggressive levels reduces VRAM usage and can make inference practical on consumer hardware. But quality degradation is not linear: it depends on the model, the quantization type, and the use domain. Public benchmarks often fail to capture these differences, because they are trained on standardized tests and because those publishing scores may have an interest in showing only the best side. That is where private benchmarks add value: they use questions and tasks reflecting the real work of the people who build them, not an abstract score.

The idea of keeping test data private and sharing only charts is more sophisticated than it seems. It reduces gaming: if nobody knows the questions, it is harder to optimize a model to pass a specific test. At the same time, sharing charts allows comparison between users without exposing sensitive data or internal benchmarks. In an enterprise context, this resembles evaluation practices with proprietary datasets, which do not end up in public leaderboards but drive real deployment decisions. It is a form of selective transparency: results circulate, tests do not.

Who wins? Those who need to choose a model for a concrete use case and cannot afford to test dozens of quantized variants. Open-weight projects could also benefit indirectly: if the community starts signaling which quantization levels maintain quality on long tasks or specific domains, maintainers get feedback that is less distorted than standardized benchmarks alone. Who loses? Those who have built visibility on token/s chasing or use throughput as their main selling point. The proposal shifts the center of gravity from pure speed to quality under constraints.

The structural signal is clear. As LLMs become local work tools, evaluation shifts from absolute performance to performance under constraints: how much quality remains after quantization, how stable the model is on long prompts, how much it costs in energy and hardware to keep it running. In the cloud, an inflated benchmark translates into a few extra dollars; on-premise, it translates into wasted VRAM, unusable models, and lost time. For those evaluating on-premise deployment, the trade-offs between post-quantization quality, throughput, and TCO are not resolved by a single leaderboard; AI-RADAR provides analytical frameworks at /llm-onpremise to read these trade-offs without reducing everything to one metric.

The proposal does not solve the trust problem: a system based on voluntary chart sharing can be slow, fragmented, and hard to verify. But it has the merit of shifting attention from noise to substance, and of reminding us that a useful LLM is not the one that generates tokens fastest, but the one that answers the right questions well.