Quantization at 1.75-bit, sparse or MoE architectures and distillation reduce the cost of running large LLMs, while million-token context models make VRAM and cache constraints central.
A detail in a pull request reignites the debate on a class of 120-billion-parameter LLMs. For teams managing self-hosted deployments, the size reshapes calculations around VRAM, quantization, and TCO, without official announcements. The signal points...
A Mixture-of-Experts training system cuts cumulative MoE-layer training time by 21.2 to 44.8 percent against SYMI by reducing memory demand in backpropagation and rebalancing retained work. LRMoE plans expert placement and retention quotas, selects t...
llama.cpp v0.8.0 adds compatibility with DFlash2 for Qwen3.8-Flash-Next. Tests on Apple Silicon M5 Max with quantized GGUFs show an average 17% decoding throughput gain over MTP, with peaks of 39% on list-like prompts and a minor prose regression. Th...
ISTA attributes the benchmark gap to Unsloth's 32K token reasoning budget, versus the 128K output tokens allowed in its own setup. The case of the 763B model with a 1M context window shows why inference cost, more than raw parameters, decides the res...
A pull request in the QwenLM/qwen-code repository has drawn attention to a possible 120-billion-parameter model. It is not an official announcement, but the 120B class touches a central issue for on-premise deployments: aggregated VRAM, quantization,...
At EMNLP 2026, T-Tech researchers presented LIMe, an attention extension that reuses key-value buffers from earlier layers. In tests with models around one billion parameters, training reached the same loss as the LLaMA baseline using 15% less comput...
The pull request in qwen-code signals that Qwen4 targets a one-million-token window. For self-hosted inference the cost is not theoretical: the key-value cache grows with tokens and fills VRAM along with weights. Qwen4-27B makes multi-GPU setups or a...
Alibaba brings Qwen4 to a one-million-token context window. The news emerges from a pull request in qwen-code: Qwen4-27B is expected tomorrow at the Apsara event in the Netherlands, while Qwen4-max is dated November 1st with October 1st as a likely p...
DiffuSpace has released Clover-35B-A3B-Preview, an open-source diffusion language model with a 35B MoE backbone, about 3B active parameters per token, and a 128K context window. The per-token active count, not total size, is the key metric for self-h...
The Ling-3.1-flash-1019 checkpoint is now on Hugging Face along with a BF16 base model for custom post-training. Total parameters remain 560B with 25B active and a 1M token context window. Reported scores move from 41 to 44 on AA Intelligence Index v...
inclusionAI updates Ling-3.1-flash: 560B total parameters, 25B active per token, and a 1M token context window. Agentic and scientific benchmarks improve. The base checkpoint is opened for research. The article analyzes implications for self-hosting ...
Nanbeige4.3-33B is not just a benchmark: it is a teacher for a distillation pipeline. For self-hosted inference, the signal is clear: cost shifts from deployment to distillation. The 3B can run on modest hardware, but the teacher requires resources. ...
Nanbeige presents the 33B model as stronger than Qwen3.8-27B on several benchmarks, but the core message is the distillation pipeline: it will feed compact versions such as a 3B. For self-hosted deployments, a 33B is VRAM-heavy; the path to a 3B chan...
ISTA-DASLab's release of GGUF quantized weights for DeepSeek-V4.1-Flash-1017 is not just a quantization milestone. It exposes the real bottleneck in local deployment: runtime integration. With llama.cpp still lacking official support and references p...
KLD is often treated as a universal metric for ranking quantized models, but it only measures the average divergence between the probability distributions of the base model and the quantized one. The number is fragile: it depends on the dataset, is e...
ISTA DASLab has released three GGUF builds of DeepSeek-V4.1-Flash-1017 with GSQ and RCO quantization, at 1.75 to 2.25 bits per weight over the 551.9 billion backbone parameters. A shared 110.6 GB Engram file stays memory-mapped on disk. The model run...
