The KV cache is becoming a central infrastructure bottleneck for agentic and long-context workloads, driving new training-free and learned sparse attention methods.
A system called phantom-kv injects a small bank of key/value tensors into a model's KV cache. The result is a per-request capability mode: load it, use it, unload it. The weights remain identical. The audit, however, highlights semantic limits and a ...
An end-to-end trainable architecture replaces rigid heuristics with dynamic contextual thresholds, reducing KV cache traffic without sacrificing quality. The 1.45 billion parameter model achieves up to 2.5x faster decoding than FlashAttention-2 on 51...
A training-free method reduces prefill cost in long-context LLMs by targeting 'mean dilution', where a block centroid hides relevant tokens. RBS-Attention combines a centroid-based branch with a rescue branch based on maximum key-block radius. On H10...
halogen 0.12.0 resolves deep-context degradation: on a Ryzen AI Max+ 395 with 128 GB, Qwen3.8-Flash-Next decode at one million tokens rises from 27.3 to 38.3 tok/s, and cold prefill drops from 21.2 to 17.9 minutes. For self-hosted LLM evaluation, pro...
Astera Labs' move shifts attention from compute to memory. The KV cache, critical for agentic workloads, becomes a physical constraint affecting parallelism, latency, and TCO. For self-hosted and on-premise deployments, memory hierarchy and data move...
As agentic systems multiply context tokens, the KV cache grows until it saturates GPU VRAM. Astera Labs is targeting this structural limit, signaling a shift in attention from compute to memory and interconnects. For those evaluating self-hosted depl...
