The local inference stack is consolidating around llama.cpp with MTP speculative decoding, MoE GPU caching, and broader model support. Deployments now span RTX GPUs, iPads with eGPUs, FPGAs, deskside Blackwell systems, and multi-node DGX Spark setups.
LemonSeed Studio runs local Qwen3.8-27B inference on iPad Pro with an external AMD GPU in a Thunderbolt enclosure. The LSE engine, identical across iPadOS, macOS, and Linux, hits 158.5 tokens/s on R9700 with DFlash2 speculative decoding and 64.4 on S...
A title and a link are enough to mark a shift in perception. The C/C++ runtime for local LLM inference is no longer confined to GitHub repositories: its public stage legitimizes self-hosted deployment as a strategic choice for TCO and data sovereignt...
A pull request in llama.cpp proposes a GPU cache for Mixture of Experts model components kept in system RAM, aiming to speed up inference when the full model doesn't fit in VRAM. The idea targets users with limited GPU memory and could shift the bott...
A pull request in the llama.cpp repository introduces support for GLM5Next MTP; a Reddit post confirms that GLM 5 Flash MTP can be run locally. For teams evaluating self-hosted deployment, the move widens the set of models available without cloud, wi...
FluidPD addresses prefill-decode imbalance in LLM serving without adding workers. FluidToken offloads part of prefill work to decode workers when slack is available; FluidRole reassigns running workers between roles without reloading the model. On Az...
Gigabyte tested the W775-V10-L01 with NVIDIA GB300, recording more than 2.2 billion tokens per day on a Blackwell Ultra GPU, 800 Gbps from a ConnectX-8 SuperNIC, and a C2C link between Grace CPU and GPU. The deskside benchmark signals that high-capac...
The local inference runtime llama.cpp reaches version 0.6.0, adding MTP speculative decoding support for Qwen4Exp models. The move strengthens the self-hosted LLM path, where hardware control and pipeline ownership matter more than API convenience.
Phoronix compares the Vulkan and ROCm back-ends of Llama.cpp on two systems with AMD Radeon GPUs and the Lemonade local AI server. The test highlights a structural choice for self-hosted AMD stacks: hardware compatibility versus optimized performance...
A GitHub project runs an INT4 Qwen3.5-9B LLM on ex-mining SQRL FPGAs. Two FK33 boards at 75 MHz deliver 3.2 tok/s generation at short context and 2.4 tok/s between 2k and 3k tokens. Estimates for Qwen3.8-27B reach 25 tok/s with four XCVU35P dies at 2...
