Experiments with Qwen, MoE models, KV cache management, expert caching, MTP, and vector indexes are making large-model inference feasible on consumer GPUs, Macs, and older datacenter hardware. The focus is shifting from raw parameter size to memory hierarchy and runtime control.
An experimental modification to llama.cpp updates the Ngram PLE table of Qwen models in memory, injecting new knowledge without reloading the model. Tested only with q8 quantization, it requires memory mapping and suffers from unreliable output contr...
The Arcon project uses Qwen3-4B with LoRA to build a local assistant with persistent memory, internal state, and tools. More than a chatbot, it is a test bed for how far a compact model can support assistant-like interaction. The answer matters for o...
The Qwen3.8-Flash-Next case on dual RTX 3090s shows that for MoE models VRAM is no longer the only variable: memory hierarchy and expert caching shift decode from 17 to 29 tokens/s. A concrete signal for teams evaluating on-premise deployments on old...
A Reddit post reports a drop from 15 hours of active programming to 4 hours for debugging or implementing a feature with Qwen 27B, even at 0.5 tokens per second. The data reframes expectations for asynchronous inference and the value of recovered hum...
For local LLMs, parameter count is not the whole constraint: the KV cache grows with every token and can become the real bottleneck at 100k or 200k context. GQA, MQA, and quantization reduce its footprint, but growth remains linear. Future optimizati...
A user documents a jump from 17 to 25-29 tokens/s on two RTX 3090s with a full 261,888-token context, using llama.cpp's LRU expert cache and a smaller ubatch. The setup keeps 48 expert layers in system RAM and shows meaningful headroom for on-prem in...
Perplexity has open-sourced the lily repository inside pplx-garden, a Mac inference server built around a single model, Qwen 3.6, to get the best performance on Apple Silicon. This is not general-purpose serving: vertical integration around one model...
Lemonade 11.9, an AMD-backed open-source local AI server, arrives on Linux, Windows, and macOS with experimental ROCm HRX backend support for Llama.cpp. The project keeps its '100% free and private' focus on GPUs, CPUs, and NPUs. The most relevant te...
Intel's LLM-Scaler-vLLM project releases a new beta based on vLLM 0.26, designed to get vLLM running on Arc and Arc Pro GPUs through Docker containers. The update lowers configuration friction for teams serving LLMs locally on Intel hardware, but it ...
The Reddit post is thin, but the message is clear: MTP support for Qwen3.8-Flash-Next in GGUF format promises to raise tokens per second in llama.cpp runs. More than a single optimization, it signals the direction of the open-source stack for local i...
Replacing the dense vocabulary projection with approximate HNSW-based search accelerates autoregressive decoding on CPU. Tested on Gemma 3, Llama 3.2 and Qwen 3, batch-one throughput improves by up to 82% for Gemma 3 270M while preserving AlpacaEval ...
Support for RadixArk/Qwen3.8-Flash-Next-NVFP4 in SGLang-V100 shows four V100 32GB running full 256k-token context with over 50GB offloaded to system RAM. Prefill around 4,000 tokens/s and decoding around 60 tokens/s. MTP data signal throughput gains ...
A Reddit setup shows a 27B dense model at 47-50 tokens/s with a 100,000-token context on a 16 GB RTX 4070 Ti SUPER. The signal is not the model but fine-grained runtime control: asymmetric kvarn5/kvarn4 KV cache, a 1,024-token full-precision tail, an...
A three-and-a-half-hour test on a 128GB MacBook Pro M5 Max ran a 79GB 2-bit Qwen3.8-Flash-Next with a 358,400-token context slot. Prefill fell from 1,561 to 318 tokens/s; generation from 30–35 to 11.5 tokens/s at 169K. After 100K tokens the model mix...
A documented Reddit setup runs a Qwen3.8-27B on an RTX 4070 Ti SUPER at 47-50 tokens/s with a 100,000-token context window. The credit goes to the beellama.cpp runtime and the asymmetric kvarn5/kvarn4 KV cache, which freed about 6% VRAM. A precision ...
