Local and Self-Hosted LLM Optimization

2026-09-05 • LocalLLaMA

Qwen3.8 27b on an RTX 3090: agentic coding goes local and reveals a gap with larger models

A technical post shows how Qwen3.8 27b in Q4_K_XL quantization runs on an RTX 3090 with 24 GB of VRAM and 100k token context. The author argues that small self-hosted LLMs capable of 80-90% of repetitive work erode cloud vendor revenue more than fron...

#Hardware #LLM On-Premise #DevOps

2026-09-05 • LocalLLaMA

Tencent Hy4 on llama.cpp: local inference shifts its center of gravity

Pull request #28127, adding Hy4-preview support to llama.cpp, brings no benchmarks but signals convergence in the self-hosted ecosystem. Even cloud vendors are paying attention to on-premise pipelines. Stability, licensing, and documentation remain o...

2026-09-04 • LocalLLaMA

Qwen3.8 27B on 16GB VRAM: benchmarking 21 quantized variants

A benchmark of 21 quantized Qwen3.8 27B variants on an RTX 5080 with 16GB VRAM shows how much GGUF choice matters. The bartowski IQ4_XS variant offers the lowest mean KL divergence and highest same-top-p agreement, while more aggressive quants prove ...

#Hardware #LLM On-Premise #DevOps

2026-09-04 • LocalLLaMA

A 90M conversational LLM runs on the Sony PSP: local AI at 2004 hardware limits

An experiment puts a 90M-parameter conversational model on a 2004 Sony console. Inference runs at about 0.5–0.6 tokens per second, with replies taking one to three minutes. More a technical provocation than a useful tool, but it says a lot about real...

#Hardware #LLM On-Premise #DevOps

2026-09-04 • LocalLLaMA

Tencent Hy4 lands on llama.cpp: a signal for local inference

Pull request #28127 adds support for the Tencent Hy4 preview architecture in llama.cpp. A concrete signal for teams evaluating on-premise stacks and data sovereignty, with no official benchmarks.

#Hardware #LLM On-Premise #DevOps

2026-09-04 • LocalLLaMA

768 GB of VRAM, €18k: the self-hosted dream hits the two-trillion-parameter wall

A builder spent €18,000 on an EPYC server with 768 GB of VRAM and 256 GB of RAM to run frontier open source LLMs locally. But upcoming models such as GLM 6 could be two to three times larger, making the machine unusable at 4-bit quantization. The cas...

#Hardware #LLM On-Premise

2026-09-04 • LocalLLaMA

Qwen 3.8 Flash Next Locally Builds a Playable FPS

A user built a multiplayer FPS with a local Qwen 3.8 Flash Next model, Q4_K_XL quantization and 256k context, using opencode. Playable demo in two hours, three days of refinement, 20 tokens/s avg with MTP on RTX 5090 + RTX 4000 PRO. It highlights the...

#Hardware #LLM On-Premise #DevOps

2026-09-03 • LocalLLaMA

IFM releases K2-Horizon-MoVA-36B-A4B in GGUF: a frontier model with 4B active parameters

IFM has published the final checkpoint of the sparse Mixture-of-Experts model K2-Horizon-MoVA-36B-A4B, with 36 billion total parameters and 4 billion active per token. Released in GGUF format, it competes on agentic and reasoning tasks with closed fr...

#Hardware #LLM On-Premise #Fine-Tuning

2026-09-03 • LocalLLaMA

Qwen-3.8-Next-Flash: Hot-Swappable Ngram Knowledge Injection in llama.cpp

An experimental modification to llama.cpp updates the Ngram PLE table of Qwen models in memory, injecting new knowledge without reloading the model. Tested only with q8 quantization, it requires memory mapping and suffers from unreliable output contr...

#LLM On-Premise #Fine-Tuning #DevOps

2026-09-03 • LocalLLaMA

Arcon pushes Qwen3-4B: how much architecture matters beyond parameter count

The Arcon project uses Qwen3-4B with LoRA to build a local assistant with persistent memory, internal state, and tools. More than a chatbot, it is a test bed for how far a compact model can support assistant-like interaction. The answer matters for o...

#LLM On-Premise #Fine-Tuning #DevOps

2026-09-03 • LocalLLaMA

Expert caching rewrites TCO: fluid MoE on aging on-premise hardware

The Qwen3.8-Flash-Next case on dual RTX 3090s shows that for MoE models VRAM is no longer the only variable: memory hierarchy and expert caching shift decode from 17 to 29 tokens/s. A concrete signal for teams evaluating on-premise deployments on old...

2026-09-03 • LocalLLaMA

Slow compute as a choice: a Qwen 27B and asynchronous productivity

A Reddit post reports a drop from 15 hours of active programming to 4 hours for debugging or implementing a feature with Qwen 27B, even at 0.5 tokens per second. The data reframes expectations for asynchronous inference and the value of recovered hum...

#Hardware #LLM On-Premise #DevOps

2026-09-03 • LocalLLaMA

Why KV cache matters more than parameter count for local long-context models

For local LLMs, parameter count is not the whole constraint: the KV cache grows with every token and can become the real bottleneck at 100k or 200k context. GQA, MQA, and quantization reduce its footprint, but growth remains linear. Future optimizati...

#Hardware #LLM On-Premise #DevOps

2026-09-03 • LocalLLaMA

Qwen3.8-Flash-Next on two RTX 3090s: expert cache pushes decode to 29 t/s

A user documents a jump from 17 to 25-29 tokens/s on two RTX 3090s with a full 261,888-token context, using llama.cpp's LRU expert cache and a smaller ubatch. The setup keeps 48 expert layers in system RAM and shows meaningful headroom for on-prem in...

#Hardware #LLM On-Premise #Fine-Tuning

2026-09-03 • LocalLLaMA

Perplexity open-sources Mac inference server optimized for Qwen 3.6

Perplexity has open-sourced the lily repository inside pplx-garden, a Mac inference server built around a single model, Qwen 3.6, to get the best performance on Apple Silicon. This is not general-purpose serving: vertical integration around one model...

#Hardware #LLM On-Premise #DevOps

2026-09-03 • Phoronix

Lemonade 11.9 Ships Experimental AMD ROCm HRX Backend via Llama.cpp

Lemonade 11.9, an AMD-backed open-source local AI server, arrives on Linux, Windows, and macOS with experimental ROCm HRX backend support for Llama.cpp. The project keeps its '100% free and private' focus on GPUs, CPUs, and NPUs. The most relevant te...

#Hardware #LLM On-Premise #DevOps

2026-09-02 • ArXiv cs.LG

REAL-Q brings gradient descent to LLM quantization

A new post-training quantization method, REAL-Q, uses block-wise gradient descent to reduce end-to-end KL divergence by up to about 49% compared with second-order methods on LLaMA-3.1 and Qwen3 at W4A16. The result matters for resource-constrained de...

#Hardware #LLM On-Premise #Fine-Tuning

2026-09-01 • LocalLLaMA

Spark-X2.5: 1M Token Context, Custom Architecture, and a Required llama.cpp Fork

Two compact LLMs, Spark-X2.5-4B and Spark-X2.5-1.7B, claim a native 1-million-token context and an architecture that is not a fine-tune. The Hugging Face page reports the 4B version rivaling a 9B Qwen 3.5 model. Local inference currently requires a c...

#LLM On-Premise #Fine-Tuning #DevOps

2026-09-01 • LocalLLaMA

MTP for Qwen3.8-Flash-Next-GGUF Promises a Throughput Boost in Local Inference

The Reddit post is thin, but the message is clear: MTP support for Qwen3.8-Flash-Next in GGUF format promises to raise tokens per second in llama.cpp runs. More than a single optimization, it signals the direction of the open-source stack for local i...

#Hardware #LLM On-Premise #DevOps

2026-08-31 • LocalLLaMA

GLM 5.3 and Flash on RTX PRO 6000 WS: BlenderMCP test exposes token and VRAM cost

A Reddit experiment runs GLM 5.3 and Flash in Q4 quantization on 4-6 RTX PRO 6000 WS to generate a penthouse in Blender via BlenderMCP. Flash completes the scene with one-third of the tokens and similar timing, while the full model spends 22 minutes ...

#Hardware #LLM On-Premise

2026-08-31 • ArXiv cs.CL

Vector Index Output Heads Speed Up CPU LLM Inference

Replacing the dense vocabulary projection with approximate HNSW-based search accelerates autoregressive decoding on CPU. Tested on Gemma 3, Llama 3.2 and Qwen 3, batch-one throughput improves by up to 82% for Gemma 3 270M while preserving AlpacaEval ...

#Hardware #LLM On-Premise #DevOps

2026-08-30 • LocalLLaMA

Uncensored GGUF wave: LongCat sparse, Qwen MTP and Laguna-S2.1 for llama.cpp

LLMFan46 has published GGUF variants of LongCat-Flash-Lite-Sparse, Qwen3.8-27B, and Qwen3.5-122B-A10B, plus Qwen3-Coder-Next and Laguna-S2.1 with optional vision. The need for a personal llama.cpp fork to run sparse attention and a 1M token context h...

#Hardware #LLM On-Premise #DevOps

2026-08-30 • LocalLLaMA

Four V100 32GB serve a 256k-token LLM: what the SGLang-V100 test shows

Support for RadixArk/Qwen3.8-Flash-Next-NVFP4 in SGLang-V100 shows four V100 32GB running full 256k-token context with over 50GB offloaded to system RAM. Prefill around 4,000 tokens/s and decoding around 60 tokens/s. MTP data signal throughput gains ...

#Hardware #LLM On-Premise #DevOps

2026-08-30 • LocalLLaMA

Qwen3.8-Flash-Next on M5 Max: 350K tokens and the limits of 2-bit

A three-and-a-half-hour test on a 128GB MacBook Pro M5 Max ran a 79GB 2-bit Qwen3.8-Flash-Next with a 358,400-token context slot. Prefill fell from 1,561 to 318 tokens/s; generation from 30–35 to 11.5 tokens/s at 169K. After 100K tokens the model mix...

#Hardware #LLM On-Premise #DevOps

2026-08-29 • LocalLLaMA

A 27B LLM on a consumer GPU: 50 tokens/s and 100k context window

A documented Reddit setup runs a Qwen3.8-27B on an RTX 4070 Ti SUPER at 47-50 tokens/s with a 100,000-token context window. The credit goes to the beellama.cpp runtime and the asymmetric kvarn5/kvarn4 KV cache, which freed about 6% VRAM. A precision ...

#Hardware #LLM On-Premise #DevOps