Open-source runtimes such as llama.cpp and OpenVINO, along with specialized engines, are accelerating local LLM inference on consumer hardware through quantization, prompt drafting and CPU/GPU optimizations.
A pull request reopens MTP integration for Qwen Flash Next in llama.cpp, with GGUF quantizations available on Hugging Face. The work is still WIP; for local LLM operators it becomes a test bed for reducing latency without hardware changes, but with a...
Pull request #27773 by timkhronos adds GLM-5.3-Flash (GLM5-Next) support to llama.cpp. The model can now run on home computers without cloud services. A signal for on-premise deployments: open source runtimes are consolidating as a distribution chann...
ISTA DASLab released quantized GGUFs for Qwen3.8-Flash-Next and a Coder variant with half the experts removed. The GSQ and RCO combination brings the resident working set to 29.6 GB, within a 32 GB accelerator, with benchmarks near or above BF16. A s...
A test on 50 MATH-500 benchmark questions shows that penalizing hesitation tokens via logit bias improves accuracy for Qwen3.5-4B in GGUF format across all quantization levels tested. BF16 goes from 74% to 84%, while Q2_K doubles from 12% to 24%. The...
The open-source runtime for local LLM inference brings a change that makes prompt lookup drafting 42 times faster. This is not full generation speed, but a key component of speculative decoding: for self-hosted deployments, it can cut overhead and TC...
A pull request in llama.cpp introduces a tiled matrix-vector multiplication for k-quant formats. According to the report, using VNNI instructions speeds up CPU mul_mat by 3-7x with minimal added complexity. The improvement targets prompt prefill, mak...
A sparse Reddit note points to Oh My Pi and custom model support through vLLM, llama.cpp, and SGLang. The real story is structural: self-hosted inference still depends on multiple runtimes with distinct hardware profiles, and integration—not a single...
An open source contribution adds OpenVINO support to Laya: response time drops to 40 milliseconds per question on CPU, 3.4 times faster than PyTorch. The result shows how runtime optimization can reduce latency and cost for lightweight inference work...
A llama.cpp Q8_0 benchmark shows both fine-tunes cut median completion tokens by about 40% compared with Qwen3.8-27B. Pass rates remain similar, but Swift delivers a more uniform reduction, while ThinkingCap has a longer tail and different language-l...
A 12GB RTX 5070 runs Qwen3.8-Flash-Next at 65.1 tokens/s with a custom inference engine. The Strata project shows how a specialized runtime can beat llama.cpp on the same numerical precision, lifting prompt processing to 543 tokens/s on a 128K contex...
A user documents running Qwen-3.8-27B locally on a Raspberry Pi with Q4_K_S quantization and Q8_0 context. Costs: 2.4 cents per 1M input tokens and 70 cents per 1M output tokens, comparable to the cheapest providers. The experience highlights the pot...
A custom llama.cpp branch brings expert expansion to MoE models and tests it on Apple Metal. The author says it beats their DS4 version, but cross-platform validation is missing. The case highlights how local inference for sparse models still depends...
A local comparison with Q8_K_XL quantization on two 128GB StrixHalo nodes shows that DeepSeek-V4-Flash-Vision generates tokens more slowly than Qwen3.8-Flash-Next, yet completes tasks in about half the time. The author observes fewer hallucinations a...
A llama.cpp pull request adds support for Spark-X2.5, two compact 4B and 1.7B parameter LLMs with a native context window of up to 1M tokens and a hybrid sliding-window attention architecture. The move lowers the barrier for local and self-hosted inf...
Eleven days and about 167 GPU hours to compare eight abliterated Qwen 3.8 27B variants against the base model. The data shows surgical edits beating aggressive ones: the most heavily edited models loop in their thinking and lose usability. Copyright ...
The Qwen3.8-Flash-Next-oQ4e-mtp variant shows local inference speeds on Apple Silicon: 45 tokens per second on M4 Max and 25 on M2 Ultra, according to llm-bench.io. The source also notes similar speed to Qwen3.8 27B on Apple Silicon. Useful for self-...
A technical post shows how Qwen3.8 27b in Q4_K_XL quantization runs on an RTX 3090 with 24 GB of VRAM and 100k token context. The author argues that small self-hosted LLMs capable of 80-90% of repetitive work erode cloud vendor revenue more than fron...
Pull request #28127, adding Hy4-preview support to llama.cpp, brings no benchmarks but signals convergence in the self-hosted ecosystem. Even cloud vendors are paying attention to on-premise pipelines. Stability, licensing, and documentation remain o...
A benchmark of 21 quantized Qwen3.8 27B variants on an RTX 5080 with 16GB VRAM shows how much GGUF choice matters. The bartowski IQ4_XS variant offers the lowest mean KL divergence and highest same-top-p agreement, while more aggressive quants prove ...
Pull request #28127 adds support for the Tencent Hy4 preview architecture in llama.cpp. A concrete signal for teams evaluating on-premise stacks and data sovereignty, with no official benchmarks.
A builder spent €18,000 on an EPYC server with 768 GB of VRAM and 256 GB of RAM to run frontier open source LLMs locally. But upcoming models such as GLM 6 could be two to three times larger, making the machine unusable at 4-bit quantization. The cas...
A user built a multiplayer FPS with a local Qwen 3.8 Flash Next model, Q4_K_XL quantization and 256k context, using opencode. Playable demo in two hours, three days of refinement, 20 tokens/s avg with MTP on RTX 5090 + RTX 4000 PRO. It highlights the...
An experimental modification to llama.cpp updates the Ngram PLE table of Qwen models in memory, injecting new knowledge without reloading the model. Tested only with q8 quantization, it requires memory mapping and suffers from unreliable output contr...
The Arcon project uses Qwen3-4B with LoRA to build a local assistant with persistent memory, internal state, and tools. More than a chatbot, it is a test bed for how far a compact model can support assistant-like interaction. The answer matters for o...
The Qwen3.8-Flash-Next case on dual RTX 3090s shows that for MoE models VRAM is no longer the only variable: memory hierarchy and expert caching shift decode from 17 to 29 tokens/s. A concrete signal for teams evaluating on-premise deployments on old...
A Reddit post reports a drop from 15 hours of active programming to 4 hours for debugging or implementing a feature with Qwen 27B, even at 0.5 tokens per second. The data reframes expectations for asynchronous inference and the value of recovered hum...
For local LLMs, parameter count is not the whole constraint: the KV cache grows with every token and can become the real bottleneck at 100k or 200k context. GQA, MQA, and quantization reduce its footprint, but growth remains linear. Future optimizati...
A user documents a jump from 17 to 25-29 tokens/s on two RTX 3090s with a full 261,888-token context, using llama.cpp's LRU expert cache and a smaller ubatch. The setup keeps 48 expert layers in system RAM and shows meaningful headroom for on-prem in...
Perplexity has open-sourced the lily repository inside pplx-garden, a Mac inference server built around a single model, Qwen 3.6, to get the best performance on Apple Silicon. This is not general-purpose serving: vertical integration around one model...
Two compact LLMs, Spark-X2.5-4B and Spark-X2.5-1.7B, claim a native 1-million-token context and an architecture that is not a fine-tune. The Hugging Face page reports the 4B version rivaling a 9B Qwen 3.5 model. Local inference currently requires a c...
The Reddit post is thin, but the message is clear: MTP support for Qwen3.8-Flash-Next in GGUF format promises to raise tokens per second in llama.cpp runs. More than a single optimization, it signals the direction of the open-source stack for local i...
A Reddit experiment runs GLM 5.3 and Flash in Q4 quantization on 4-6 RTX PRO 6000 WS to generate a penthouse in Blender via BlenderMCP. Flash completes the scene with one-third of the tokens and similar timing, while the full model spends 22 minutes ...
Replacing the dense vocabulary projection with approximate HNSW-based search accelerates autoregressive decoding on CPU. Tested on Gemma 3, Llama 3.2 and Qwen 3, batch-one throughput improves by up to 82% for Gemma 3 270M while preserving AlpacaEval ...
LLMFan46 has published GGUF variants of LongCat-Flash-Lite-Sparse, Qwen3.8-27B, and Qwen3.5-122B-A10B, plus Qwen3-Coder-Next and Laguna-S2.1 with optional vision. The need for a personal llama.cpp fork to run sparse attention and a 1M token context h...
Support for RadixArk/Qwen3.8-Flash-Next-NVFP4 in SGLang-V100 shows four V100 32GB running full 256k-token context with over 50GB offloaded to system RAM. Prefill around 4,000 tokens/s and decoding around 60 tokens/s. MTP data signal throughput gains ...
A Reddit setup shows a 27B dense model at 47-50 tokens/s with a 100,000-token context on a 16 GB RTX 4070 Ti SUPER. The signal is not the model but fine-grained runtime control: asymmetric kvarn5/kvarn4 KV cache, a 1,024-token full-precision tail, an...
A three-and-a-half-hour test on a 128GB MacBook Pro M5 Max ran a 79GB 2-bit Qwen3.8-Flash-Next with a 358,400-token context slot. Prefill fell from 1,561 to 318 tokens/s; generation from 30–35 to 11.5 tokens/s at 169K. After 100K tokens the model mix...
A documented Reddit setup runs a Qwen3.8-27B on an RTX 4070 Ti SUPER at 47-50 tokens/s with a 100,000-token context window. The credit goes to the beellama.cpp runtime and the asymmetric kvarn5/kvarn4 KV cache, which freed about 6% VRAM. A precision ...
llama.cpp has merged support for Qwen3.8-Flash-Next and a community report shows 55 tokens/s with a Q4 GGUF on four RTX 3090s. An informal test that shifts attention from software to hardware: for local inference, multi-GPU cost and complexity remain...
A Mac Studio M4 Max with 128GB of unified memory hosted the test of Qwen3.8-Flash-Next, the first model this year to break 94% on tolitius's personal benchmark. The 27B model excels at coding but loses in general knowledge to Gemma 31B and Qwen 3.6 o...
