Topic / Trend Rising

Self-Hosted and Local LLM Inference

llama.cpp reaches 1.0 and AMD Loom brings 50 LLMs to client GPUs, while Unsloth and AMD enable Ryzen AI NPU acceleration. Open tools like Inside-llm and compact MoE models such as OVH Olala Mini lower the barrier to self-hosting on consumer hardware.

Detected: 2026-11-20 ยท Updated: 2026-11-20

Related Coverage

2026-11-19 โ€ข Phoronix

Unsloth Desktop now uses AMD Ryzen AI NPUs, including on Linux

AMD and Unsloth have announced local Inference support for Ryzen AI NPUs in Unsloth Desktop, starting with XDNA 2 and including Linux compatibility. The move reduces dependence on discrete GPUs and cloud services for lighter workloads, with potential...

#Hardware #LLM On-Premise #Fine-Tuning
2026-11-17 โ€ข LocalLLaMA

OVH launches Olala Mini 7A1B: an open-source LLM that runs on 8 GB of VRAM

OVH has unveiled Olala Mini, an open-source LLM with 7 billion parameters and 1 billion active, capable of running on an 8 GB GPU and even on CPU. Built on a sparse mixture-of-experts architecture with hybrid attention, it reaches 160 tokens/s on an ...

#Hardware #LLM On-Premise #Fine-Tuning
2026-11-15 โ€ข LocalLLaMA

Inside-llm: seeing inside an LLM and comparing two quantizations token by token

A new open-source tool observes in real time what happens inside an LLM and compares two quantizations of the same model token by token. Inside-llm uses llama.cpp and GGUF, runs locally, and displays probabilities, divergence, the sampler chain, and ...

#Hardware #LLM On-Premise #Fine-Tuning
2026-11-15 โ€ข LocalLLaMA

Llama.cpp catches up: local benchmarks on RTX 5080 show a turning point

A hobbyist compared llama.cpp over time with his own inference engine on an RTX 5080 16GB. Within a few months llama.cpp went from being slower to surpassing the custom alternative, while keeping efficient VRAM usage. The story signals a maturation o...

#Hardware #LLM On-Premise #DevOps
2026-11-14 โ€ข Phoronix

LLM porting as a TCO variable: the signal from AMD

AMD's HRX Loom and Ryzen AI Max experiment shows that moving 50 LLMs to client GPUs in three days is not just a hardware feat. The real signal is the shift of value toward middleware and porting pipelines, with effects on self-hosted setups, data sov...

2026-11-13 โ€ข Phoronix

AMD: 50 LLMs on client GPUs in three days with HRX Loom

An AMD team ported 50 trending Hugging Face models to client GPUs in about three days, using an AI agent workflow and HRX Loom v0.1 software on Ryzen AI Max 'Strix Halo' hardware. The experiment signals that the barrier to local inference is no longe...

#Hardware #LLM On-Premise #DevOps
2026-11-13 โ€ข LocalLLaMA

llama.cpp reaches version 1.0 and becomes a platform for local AI

llama.cpp releases version 1.0: the runtime for running LLMs locally becomes a stable foundation for downstream applications. Hugging Face also announces desktop apps to reduce local AI access to a few clicks. A maturity step that strengthens the sel...

#Hardware #LLM On-Premise #DevOps
โ† Back to All Topics