1-bit weights, dynamic GGUF, KV cache quantization, and multi-token prediction drive more efficient local inference. New benchmark proposals emphasize reproducibility instead of raw tokens per second.
A benchmark across 35 weight quantizations with llama.cpp, ExLlamaV3 and vLLM measures the KL divergence introduced by quantized KV caches. At 8-bit, memory drops by about 1.9-2 GiB at 64k tokens and the effect on token distribution is nearly zero. 4...
An anecdotal report about a Q4 model running with llama.cpp on an RTX 4090 reopens the reproducibility problem in self-hosted deployments: without hashes and pinned versions, local control remains incomplete.
The November 10 deadline for PyTorch Day China 2026 reveals an agenda focused on distributed training, efficient inference, hardware acceleration, and physical AI. The analysis: the chosen tracks show the framework increasingly looking at production ...
The PyTorch Day China 2026 Call for Proposals closes November 10. The December 12 event in Beijing highlights five topic areas, with a notable emphasis on efficient inference, hardware acceleration, and physical AI. For teams evaluating local stacks ...
Unsloth has released new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-Flash-Next, with 10-15% accuracy gains on agentic benchmarks such as DeepSWE and Terminal Bench. It includes a switching process between BF16, Q8_0, Q5_1, and IQ4_NL precisions, with dec...
A Reddit post turns a llama.cpp fork into a formula with square brackets: ultra-specific quantization and hardware produce incredible but irreproducible results. The case exposes the fragile side of local benchmarks and what it means for those evalua...
ISTA DASLab has released a Coder build of DeepSeek-V4.1-Flash-1017 with half its routed experts removed and transformer weights compressed to 1 bit per parameter. The GGUF takes 72.1 GB of transformer weights, down from 138.2 GB unpruned, and retains...
A LocalLLaMA user proposes moving beyond token/s chasing and building private quality benchmarks, especially across quantization levels. The idea: keep test data private and share only comparative charts. It is a reaction to benchmark noise and score...
The JetBrains/Mellum2.1-12B-A2.5B-Thinking-MTP repository on Hugging Face introduces an integrated multi-token prediction head that accelerates decoding by 1.4–1.8x while accepting about 90% of proposed tokens. vLLM 0.33, transformers 5.20 and llama....
