The local inference community is looking beyond token/s metrics, focusing on quality benchmarks, quantization trade-offs, and decoding acceleration. Irreproducible results and regressions highlight the need for locked versions and rigorous testing.
An anecdotal report about a Q4 model running with llama.cpp on an RTX 4090 reopens the reproducibility problem in self-hosted deployments: without hashes and pinned versions, local control remains incomplete.
A Reddit post turns a llama.cpp fork into a formula with square brackets: ultra-specific quantization and hardware produce incredible but irreproducible results. The case exposes the fragile side of local benchmarks and what it means for those evalua...
ISTA DASLab has released a Coder build of DeepSeek-V4.1-Flash-1017 with half its routed experts removed and transformer weights compressed to 1 bit per parameter. The GGUF takes 72.1 GB of transformer weights, down from 138.2 GB unpruned, and retains...
A LocalLLaMA user proposes moving beyond token/s chasing and building private quality benchmarks, especially across quantization levels. The idea: keep test data private and share only comparative charts. It is a reaction to benchmark noise and score...
The JetBrains/Mellum2.1-12B-A2.5B-Thinking-MTP repository on Hugging Face introduces an integrated multi-token prediction head that accelerates decoding by 1.4–1.8x while accepting about 90% of proposed tokens. vLLM 0.33, transformers 5.20 and llama....
