New compact, sparse or quantized open models, plus runtime and reasoning optimizations, are making high-context LLM deployment viable on consumer GPUs, CPU and self-hosted setups.
A new post-training quantization method, REAL-Q, uses block-wise gradient descent to reduce end-to-end KL divergence by up to about 49% compared with second-order methods on LLaMA-3.1 and Qwen3 at W4A16. The result matters for resource-constrained de...
Two compact LLMs, Spark-X2.5-4B and Spark-X2.5-1.7B, claim a native 1-million-token context and an architecture that is not a fine-tune. The Hugging Face page reports the 4B version rivaling a 9B Qwen 3.5 model. Local inference currently requires a c...
The Reddit post is thin, but the message is clear: MTP support for Qwen3.8-Flash-Next in GGUF format promises to raise tokens per second in llama.cpp runs. More than a single optimization, it signals the direction of the open-source stack for local i...
ERR+ introduces a two-phase RLVR framework that rewards token-level entropy drops and relative length efficiency. Correct traces show more frequent and larger entropy drops during thinking. Tests on five datasets report improved accuracy and concisen...
A Reddit experiment runs GLM 5.3 and Flash in Q4 quantization on 4-6 RTX PRO 6000 WS to generate a penthouse in Blender via BlenderMCP. Flash completes the scene with one-third of the tokens and similar timing, while the full model spends 22 minutes ...
A Hugging Face repository named DeepSeek-V4-Flash-Vision-Exp is circulating on Reddit without technical details. The name suggests an efficient, experimental vision-language model, but without VRAM, quantization, or license. For those evaluating self...
Replacing the dense vocabulary projection with approximate HNSW-based search accelerates autoregressive decoding on CPU. Tested on Gemma 3, Llama 3.2 and Qwen 3, batch-one throughput improves by up to 82% for Gemma 3 270M while preserving AlpacaEval ...
LLMFan46 has published GGUF variants of LongCat-Flash-Lite-Sparse, Qwen3.8-27B, and Qwen3.5-122B-A10B, plus Qwen3-Coder-Next and Laguna-S2.1 with optional vision. The need for a personal llama.cpp fork to run sparse attention and a 1M token context h...
Support for RadixArk/Qwen3.8-Flash-Next-NVFP4 in SGLang-V100 shows four V100 32GB running full 256k-token context with over 50GB offloaded to system RAM. Prefill around 4,000 tokens/s and decoding around 60 tokens/s. MTP data signal throughput gains ...
A Reddit setup shows a 27B dense model at 47-50 tokens/s with a 100,000-token context on a 16 GB RTX 4070 Ti SUPER. The signal is not the model but fine-grained runtime control: asymmetric kvarn5/kvarn4 KV cache, a 1,024-token full-precision tail, an...
A three-and-a-half-hour test on a 128GB MacBook Pro M5 Max ran a 79GB 2-bit Qwen3.8-Flash-Next with a 358,400-token context slot. Prefill fell from 1,561 to 318 tokens/s; generation from 30–35 to 11.5 tokens/s at 169K. After 100K tokens the model mix...
A documented Reddit setup runs a Qwen3.8-27B on an RTX 4070 Ti SUPER at 47-50 tokens/s with a 100,000-token context window. The credit goes to the beellama.cpp runtime and the asymmetric kvarn5/kvarn4 KV cache, which freed about 6% VRAM. A precision ...
Z.ai uses the same base model as GLM-5.2 for GLM-5.3, but all gains come from post-training. Coding improves by 50% on the internal Code Bench and reaches state of the art on Terminal Bench 3.0 and Agents' Last Exam. The real shift is the emergent cy...
llama.cpp has merged support for Qwen3.8-Flash-Next and a community report shows 55 tokens/s with a Q4 GGUF on four RTX 3090s. An informal test that shifts attention from software to hardware: for local inference, multi-GPU cost and complexity remain...
The Apodex team's AMA on r/LocalLLaMA introduces Apodex 1.1, an open model family for agentic work, plus an open-source harness and two papers. Quantized variants in FP8, GPTQ-Int4, and NVFP4 point to local, self-hosted inference, but no benchmark fi...
The Apodex team has presented Apodex 1.1, an open model family designed for complex work involving reasoning, search, code execution, and multi-agent coordination. Alongside the models, the team released FrontierAgent and two papers. The NVFP4, GPTQ-...
A Mac Studio M4 Max with 128GB of unified memory hosted the test of Qwen3.8-Flash-Next, the first model this year to break 94% on tolitius's personal benchmark. The 27B model excels at coding but loses in general knowledge to Gemma 31B and Qwen 3.6 o...
The Hugging Face page for zai-org/GLM-5.3-Flash signals a new model, but no technical details. For self-hosted deployments, the Flash label suggests inference efficiency while missing VRAM, quantization, and token details complicate planning. We anal...
The Qwen team has released Qwen3.8-Flash-Next, a 125B open-weight model with 6B active parameters and a native 262,144-token context. The architecture combines Qwen Sparse Attention, Gated Residual, and a 51B n-gram embedding to reduce long-context l...
IBM has released Granite 4.2, three open-weight LLMs with 3, 8 and 30 billion parameters designed for download and self-hosting. All provide a native 128,000-token context window. The 8B and 30B variants add an agentic reinforcement learning block fo...
