New training-free quantization, sparse attention, KV-cache modulation, and external memory techniques are reducing LLM inference costs and long-context overhead. Work also focuses on compact models and controlled reasoning length to make local deployment more predictable.
Qwengram-0.8B freezes Qwen3.5-0.8B and adds a 51-billion-parameter n-gram memory outside the GGUF. Perplexity drops by 5.05%, but the relevant signal is architectural: late injection controlled by a per-token gate, Q8_0 quantized sidecar, and no back...
A developer trained a compact model to replace Gemini Flash in a high-frequency boolean task. Using 2,500 examples and a 16 GB RTX A6000, training took 15 minutes: the result weighs 50 MB, runs on CPU in 0.06 seconds, and reaches 97% accuracy. A conc...
An independent benchmark compares Jev and Kev 4B on 362 fresh items: similar accuracy, but Jev adds 257 fixed input tokens per request, raising cost up to 12x on short prompts.
An experiment keeps Qwen3.5-0.8B frozen, adds a 51-billion-parameter PLE memory and an R=1 reader: validation perplexity drops by 5.05%. The author warns this is not benchmark accuracy gain and releases GGUFs, a modified llama.cpp runtime, and a quan...
A llama.cpp Q8_0 benchmark shows both fine-tunes cut median completion tokens by about 40% compared with Qwen3.8-27B. Pass rates remain similar, but Swift delivers a more uniform reduction, while ThinkingCap has a longer tail and different language-l...
PRQuant combines channel reorganization and static residual compensation to reduce inference overhead in low-bit quantization. The method is training-free, builds contiguous tail blocks offline, and avoids dynamic online gathering by turning scattere...
A system called phantom-kv injects a small bank of key/value tensors into a model's KV cache. The result is a per-request capability mode: load it, use it, unload it. The weights remain identical. The audit, however, highlights semantic limits and a ...
An end-to-end trainable architecture replaces rigid heuristics with dynamic contextual thresholds, reducing KV cache traffic without sacrificing quality. The 1.45 billion parameter model achieves up to 2.5x faster decoding than FlashAttention-2 on 51...
A training-free method reduces prefill cost in long-context LLMs by targeting 'mean dilution', where a block centroid hides relevant tokens. RBS-Attention combines a centroid-based branch with a rescue branch based on maximum key-block radius. On H10...
A post on r/ROCm highlights the problem of FP4-optimized inference engines: fast in benchmarks, but often unreliable on small dense models. Compression can work on Large Language Models with many parameters, but on small models logical errors become ...
