Hardware for Local Intelligence
Benchmarks, GPU sizing guides, and workstation builds for sovereignty.
Two numbers decide almost everything in local-LLM hardware: VRAM (whether the model fits at all)
and memory bandwidth (your tokens/sec — single-user generation streams the whole model through the GPU
per token, so GB/s matters more than TFLOPS). The rule of thumb:
VRAM ≈ params(B) × 0.55 × 1.15 at 4-bit — a 34B fits a 24GB card, a 70B wants 48GB.
Long contexts add KV-cache on top, and it can rival the weights at 100k+ tokens.
The practical tiers: 24GB (used RTX 3090 / 4090 — the value sweet spot, runs 7–34B), 32GB (RTX 5090 — fastest consumer inference), 48GB (RTX A6000-class or dual 24GB — the 70B tier), 80GB+ (datacenter cards — long context, 8-bit, training). Apple Silicon plays a different game: huge unified memory holds models no consumer GPU can, at lower speeds.
Deep references: GPU buyer guide · VRAM sizing math · Hardware Matrix · quantization. Below: the latest hardware intelligence from the radar.
FastFlowLM 1.0: AMD brings NPU AI under the ROCm umbrella
FastFlowLM 1.0, the open-source software for running language and multimodal models on Ryzen AI NPUs, officially joins the ROCm ecosystem. The...
Luth-2: French small language models beat 3x larger competitors, redefining local AI
The new Luth-2-0.8B and 2B models set a new state of the art for French, outperforming models up to three times their size across language and...
Self-adaptive fuzzing exposes the hallucination cracks in multimodal LLMs
A new evaluation framework pairs a unified taxonomy benchmark with self-adaptive multimodal fuzzing (SAMF) and shows that state-of-the-art MLLMs...
Training a 1B LLM from scratch for under $200: the frontier of accessible self-hosting
A personal project demonstrates how to train a 1.1-billion-parameter LLM from scratch on 20 billion tokens for around $200. The model, fine-tuned...
Linux: Open-Source NVIDIA “Nova” Driver Gains Functionality with Rust
The Nova DRM driver, successor to Nouveau, gets new features for Linux 7.3 kernel. Written in Rust, it aims to provide open-source support for...
Fedora CoreOS enables systemd-oomd and zRAM swap: a signal for on-premise inference
Starting with release 45, Fedora's container-optimized OS will handle memory pressure automatically via systemd-oomd and zRAM-backed swap. A move...
Ling-3.0-tiny: 8B parameters, 1.3B active, hitting 100 tokens/sec on MacBook
InclusionAI releases Ling-3.0-tiny, an 8B-parameter MoE with just 1.3B active, reaching 100-105 tokens/s on DGX Spark and 86-90 on an M4 Pro...
On-device agentic AI: Meta brings Muse Glimmer to ExecuTorch with speculative decoding and 128K context
Meta releases Muse Glimmer, a 30-billion-parameter model optimized for on-device agentic workflows. ExecuTorch now supports inference on NVIDIA...
Muse-Glimmer-30B in GGUF: The Latest Piece of an Increasingly Mature Local Ecosystem
The arrival of Muse-Glimmer-30B in GGUF format isn't just a model release. It signals a strengthening local inference infrastructure, shifting...
Meta Muse Glimmer: Local AI Agents on Consumer GPUs, 30B Parameter Model Released
Meta releases Muse Glimmer, a 30-billion-parameter LLM under Apache 2.0 license, optimized for local AI agents on consumer GPUs. Quantized to...
Meta's Muse Glimmer: A 30B Open-Weight Model Purpose-Built for Always-On Local Agents
Meta releases Muse Glimmer, a 30B parameter model under Apache 2.0, optimized for always-on agentic workflows on consumer hardware. 4-bit...
Linux 7.3 removes old SGI drivers: security and AI agent noise drive kernel pruning
The kernel's removal of legacy Silicon Graphics drivers—mirrored by a surge of AI-generated patches—marks a shift in maintenance philosophy. For...
No cloud, just edge: Edgify raises $9M for AI that learns on devices
The Israeli startup closes a Series A+ round to expand its edge AI platform from grocery retail to fast food, logistics, and industry. The goal:...
TEXAS Leverages Native MoE Routing for More Surgical Fine-Tuning
A new approach called TEXAS exploits how mixture-of-experts models activate their sub-models to steer fine-tuning, focusing supervision only on...
Truth is a vector: detecting fake news without leaving the model
A research team uses activation engineering to extract a falsehood direction in the latent space of transformers. Without fine‑tuning or external...
WeatherNext 2: DeepMind brings cyclone forecasting to a single H100 GPU
An open model from DeepMind, published in Nature, improves cyclone forecasts by an extra day, but the real surprise is that it runs on a single...
BDH: Pathway's post-transformer architecture matches GPT-2 scaling on ordinary GPUs
Pathway introduces BDH, a post-transformer architecture matching GPT-2 scaling from 10 million to 1 billion parameters while training on ordinary...
Two flags take Ling-3.0-flash INT4 from 20.8 to 38.7 tok/s on a single DGX Spark
The official INT4 of Ling-3.0-flash runs on a single DGX Spark, but the default configuration delivers only 20.8 tok/s. Two changes — dropping...
Lophius: A Workbench for Transformer Research, Born from the Creator of Heretic
The fruit of two years of development: a hybrid code/GUI environment inside notebooks for inspecting models, managing prompts, analyzing logits,...
Bare die and 3D-printed block push RTX 2060 Super to 28°C under load
A modder directly cooled an RTX 2060 Super's silicon using a 3D-printed water block. Despite initial leaks, load temperatures plummeted to 28°C....
Meetily transcribes and summarizes meetings with no subscription, open source. Here’s how
A free, open-source alternative to paid transcription services: Meetily records and transcribes meetings locally, without sending sensitive data...
When AI writes drivers: DeepSeek generates a custom Metal kernel for Kimi K2 on Mac
In 50 minutes, an LLM wrote the code to run a 90GB quantized model on a Mac Studio with no existing kernel. Modest performance, but enough to show...
Kimi K3 slims to 478GB: the multilingual trim that shifts on-prem math
A Hugging Face user has released a quantized Kimi K3 variant that strips multilingual support and applies IQ2-XXS via Unsloth, shrinking the...
BitNet hits 36 tok/s on Xeon with a zero-dependency C engine—and crashes into the DRAM ceiling
A C99 inference engine with zero dependencies drives the 2-billion-parameter ternary BitNet model to 36.25 tokens per second on an Intel Xeon CPU,...
Running Kimi K3 Locally: Clusters, Aggressive Quantization, and the Faith in Smaller Models
A user runs Kimi K3 across two clusters using llama.cpp over RPC, with partial offloading and IQ1_M quantization, aiming for a 2-3x speedup by...
Radical minimalism: a coding agent in 9 lines of Python with zero dependencies
An open-source experiment demonstrates an essential coding agent: 9 lines of Python, standard library only, no external dependencies. It works...
Qwen3.6 on a Single Radeon R9700: 262K Tokens with a 32GB GPU
A power user pushes a 32GB AMD Radeon AI Pro R9700 to its limits with INT4-quantized Qwen3.6 models. The 35B MoE achieves 262,144 token context...
HWMON: Critical Bugs in Linux 7.2-rc7 Unearthed by an LLM Agent
Linux kernel 7.2-rc7 brings dozens of fixes to the hardware monitoring subsystem, mostly critical or high-severity bugs spotted by the Sashiko...
The Buzz Around Qwen 3.8: Home LLMs Are Gaining Momentum
A user’s hands-on with Qwen 3.6 27B at Q4 quantization on an Apple M5 chip sparks a vision of home-hosted LLMs that intercept queries before they...
MoE vs dense: Qwen 35B-A3B hits 116 tok/s on AMD GPU, the quality gap is surprisingly small
A local test on AMD hardware compared Qwen 35B-A3B (MoE, 3B active) against a dense 27B model. The MoE managed ~116 tok/s vs ~30 tok/s for the...
Looking for general AI news?
< AI-RADAR MAIN