Local LLM deployments advance through quantization, MoE expert caching, KV cache optimization, and hardware-specific tuning across Apple Silicon, RTX, AMD and Intel Arc. New compact open-weight models and llama.cpp integrations make on-premise inference faster, cheaper and more capable.
A developer releases Jenny, an open source Electron app for running LLMs locally with tool calling, rollback and an IDE. No network calls beyond the local runtime; safeguards on destructive commands and file edits. Windows first, with reports of soli...
A guide shows how to connect llama.cpp, the MCP protocol and FreeCAD to generate solids with a local model. The workflow uses Qwen3.8-27B quantized Q4_K_M and an mmproj-F16 vision projector: the model can call tools, read screenshots and verify geome...
A custom llama.cpp branch brings expert expansion to MoE models and tests it on Apple Metal. The author says it beats their DS4 version, but cross-platform validation is missing. The case highlights how local inference for sparse models still depends...
A local comparison with Q8_K_XL quantization on two 128GB StrixHalo nodes shows that DeepSeek-V4-Flash-Vision generates tokens more slowly than Qwen3.8-Flash-Next, yet completes tasks in about half the time. The author observes fewer hallucinations a...
A llama.cpp pull request adds support for Spark-X2.5, two compact 4B and 1.7B parameter LLMs with a native context window of up to 1M tokens and a hybrid sliding-window attention architecture. The move lowers the barrier for local and self-hosted inf...
An alternative to distillation and fine-tuning uses Codex and GPT Astra to transfer operational procedures for 3D modeling to Qwen Next via OpenCode and MCP Blender. The open model executes tasks after being guided, while the frontier model burns Pro...
Eleven days and about 167 GPU hours to compare eight abliterated Qwen 3.8 27B variants against the base model. The data shows surgical edits beating aggressive ones: the most heavily edited models loop in their thinking and lose usability. Copyright ...
The Qwen3.8-Flash-Next-oQ4e-mtp variant shows local inference speeds on Apple Silicon: 45 tokens per second on M4 Max and 25 on M2 Ultra, according to llm-bench.io. The source also notes similar speed to Qwen3.8 27B on Apple Silicon. Useful for self-...
A technical post shows how Qwen3.8 27b in Q4_K_XL quantization runs on an RTX 3090 with 24 GB of VRAM and 100k token context. The author argues that small self-hosted LLMs capable of 80-90% of repetitive work erode cloud vendor revenue more than fron...
Pull request #28127, adding Hy4-preview support to llama.cpp, brings no benchmarks but signals convergence in the self-hosted ecosystem. Even cloud vendors are paying attention to on-premise pipelines. Stability, licensing, and documentation remain o...
A benchmark of 21 quantized Qwen3.8 27B variants on an RTX 5080 with 16GB VRAM shows how much GGUF choice matters. The bartowski IQ4_XS variant offers the lowest mean KL divergence and highest same-top-p agreement, while more aggressive quants prove ...
Pull request #28127 adds support for the Tencent Hy4 preview architecture in llama.cpp. A concrete signal for teams evaluating on-premise stacks and data sovereignty, with no official benchmarks.
A builder spent €18,000 on an EPYC server with 768 GB of VRAM and 256 GB of RAM to run frontier open source LLMs locally. But upcoming models such as GLM 6 could be two to three times larger, making the machine unusable at 4-bit quantization. The cas...
A user built a multiplayer FPS with a local Qwen 3.8 Flash Next model, Q4_K_XL quantization and 256k context, using opencode. Playable demo in two hours, three days of refinement, 20 tokens/s avg with MTP on RTX 5090 + RTX 4000 PRO. It highlights the...
IFM has published the final checkpoint of the sparse Mixture-of-Experts model K2-Horizon-MoVA-36B-A4B, with 36 billion total parameters and 4 billion active per token. Released in GGUF format, it competes on agentic and reasoning tasks with closed fr...
An experimental modification to llama.cpp updates the Ngram PLE table of Qwen models in memory, injecting new knowledge without reloading the model. Tested only with q8 quantization, it requires memory mapping and suffers from unreliable output contr...
The Arcon project uses Qwen3-4B with LoRA to build a local assistant with persistent memory, internal state, and tools. More than a chatbot, it is a test bed for how far a compact model can support assistant-like interaction. The answer matters for o...
The Qwen3.8-Flash-Next case on dual RTX 3090s shows that for MoE models VRAM is no longer the only variable: memory hierarchy and expert caching shift decode from 17 to 29 tokens/s. A concrete signal for teams evaluating on-premise deployments on old...
For local LLMs, parameter count is not the whole constraint: the KV cache grows with every token and can become the real bottleneck at 100k or 200k context. GQA, MQA, and quantization reduce its footprint, but growth remains linear. Future optimizati...
A user documents a jump from 17 to 25-29 tokens/s on two RTX 3090s with a full 261,888-token context, using llama.cpp's LRU expert cache and a smaller ubatch. The setup keeps 48 expert layers in system RAM and shows meaningful headroom for on-prem in...
Perplexity has open-sourced the lily repository inside pplx-garden, a Mac inference server built around a single model, Qwen 3.6, to get the best performance on Apple Silicon. This is not general-purpose serving: vertical integration around one model...
Lemonade 11.9, an AMD-backed open-source local AI server, arrives on Linux, Windows, and macOS with experimental ROCm HRX backend support for Llama.cpp. The project keeps its '100% free and private' focus on GPUs, CPUs, and NPUs. The most relevant te...
Intel's LLM-Scaler-vLLM project releases a new beta based on vLLM 0.26, designed to get vLLM running on Arc and Arc Pro GPUs through Docker containers. The update lowers configuration friction for teams serving LLMs locally on Intel hardware, but it ...
Two compact LLMs, Spark-X2.5-4B and Spark-X2.5-1.7B, claim a native 1-million-token context and an architecture that is not a fine-tune. The Hugging Face page reports the 4B version rivaling a 9B Qwen 3.5 model. Local inference currently requires a c...
