Across the stack, inference is being optimized via speculative decoding, k-quant CPU matrix kernels, GEMM autotuning in vLLM, low-rank attention alternatives, and test-time compute loops. These changes reduce FLOPs, VRAM, and latency for smaller or cheaper deployments.
Helion's linear backend in vLLM shows a single GEMM kernel with an ahead-of-time autotuner can cover specialized variants and improve inference by up to 10% on Qwen3. For self-hosted LLM teams, the signal is twofold: less dependence on deep CUDA skil...
Red Hat and Meta integrated Helion into vLLM's linear backend. A single GEMM implementation covers Standard, Split-K, and Swap-AB variants, with per-shape autotuning on NVIDIA Hopper GPUs. On H100 tests, quantized kernels outperform CUTLASS, FlashInf...
Meta's GEM attention kernel, rewritten in TLX, beats FlashAttention-4 on jagged Blackwell workloads: +13% forward and +50% backward on average, with about 3,200 lines versus 10,000. The result shows explicit hardware control no longer requires hand-w...
DASA uses activation gradients from a frozen reference model to produce synthetic embeddings, fine-tuning LLMs without readable text. Tests on Llama and Qwen show performance similar to or better than natural language data, with up to 4.9x speedup ov...
A new method, DASA, generates synthetic fine-tuning data as continuous embeddings driven by activation gradients from a frozen reference model. In tests on six Llama and Qwen models with LoRA, it matches or surpasses natural-language data and speeds ...
Nemotron 3's IOI 2026 score is not just a milestone. The stated result rests on GenCorrect, a test-time compute loop that multiplies tokens per query. For on-premise deployments, open weights and NVFP4 quantization lower the hardware barrier, but ope...
NVIDIA's model card for Nemotron-Labs-3-Competitive-Coding describes a 550B-A55B model fine-tuned on 477,642 synthetic reasoning traces distilled from GLM-5.2. Combined with the GenCorrect strategy, it scored 535.4 out of 600 on the IOI 2026 problem ...
A new approach replaces attention in Transformers with bottleneck autoencoder modules plus iterative refinement at masked positions. In tests with BERT and C4 it reaches a significant share of attention performance with about 1.9× fewer FLOPs and mat...
A test on 50 MATH-500 benchmark questions shows that penalizing hesitation tokens via logit bias improves accuracy for Qwen3.5-4B in GGUF format across all quantization levels tested. BF16 goes from 74% to 84%, while Q2_K doubles from 12% to 24%. The...
The open-source runtime for local LLM inference brings a change that makes prompt lookup drafting 42 times faster. This is not full generation speed, but a key component of speculative decoding: for self-hosted deployments, it can cut overhead and TC...
A pull request in llama.cpp introduces a tiled matrix-vector multiplication for k-quant formats. According to the report, using VNNI instructions speeds up CPU mul_mat by 3-7x with minimal added complexity. The improvement targets prompt prefill, mak...
