Quantization, compact open-weight models, and consumer GPU/CPU optimizations are making local LLM inference practical under tight memory constraints. Releases now target sub-16GB VRAM and even CPU-only deployments.
Hugging Face has added native GGUF support to Transformers. Users can load a quantized model by specifying the file path and use standard APIs. On supported Apple Silicon, ggml kernels are reused, with M2 Max benchmarks close to llama.cpp. The goal i...
PRQuant combines channel reorganization and static residual compensation to reduce inference overhead in low-bit quantization. The method is training-free, builds contiguous tail blocks offline, and avoids dynamic online gathering by turning scattere...
SupraLabs has released Supra2-IMG, a 100M parameter DiT text-to-image model trained from scratch in under 10 hours on a single H100. It generates 256×256 images with competitive quality and runs locally: about 20 seconds on CPU and 2 on GPU. A concre...
Most local LLM users do not run multi-GPU setups: 16GB of VRAM, and often 12GB, are the real ceiling. The last six months have made agentic coding practical on this tier, for example with Qwen 27B quants. The constraint shifts attention to quantizati...
A post on r/ROCm highlights the problem of FP4-optimized inference engines: fast in benchmarks, but often unreliable on small dense models. Compression can work on Large Language Models with many parameters, but on small models logical errors become ...
halogen 0.12.0 resolves deep-context degradation: on a Ryzen AI Max+ 395 with 128 GB, Qwen3.8-Flash-Next decode at one million tokens rises from 27.3 to 38.3 tok/s, and cold prefill drops from 21.2 to 17.9 minutes. For self-hosted LLM evaluation, pro...
Von, an open-source model with 395 million parameters, is introduced as a drop-in replacement for TypeSafe's JEV. It runs entirely on CPU with 1–2 GB of memory and latency of 25–300 ms; the author says it beats JEV in all benchmarks. Code is on GitHu...
Ternary Bonsai 2 is available on Hugging Face: it derives from Qwen3.8-27B, keeps the hybrid-attention architecture, and uses ternary weights to shrink below 6GB. According to the model card, it is 9x smaller than FP16 while retaining 98.2% of the in...
UkisAI's open-source model passes 100k downloads and ranks first among fine-tunes on Hugging Face. It penalizes overthinking patterns in small LLMs, cutting token usage by 58.3% and boosting speed 1.95x without accuracy loss. Swift1.5 and Swift Qwen3...
New optimizations for single AMD Radeon R9700 cards double performance with Qwen3.8 27B in NVFP4 format: up to 153.1 tokens/s in decode, 3,619 tokens/s in prefill, and 471 aggregate tokens/s with eight concurrent requests. The updated vLLM MXFP4 repo...
