New architectures, quantization paths, MoE routing, and local training tools are reducing the cost and hardware requirements for running LLMs. Projects show that small or optimized models can match larger ones on accessible hardware.
AMD updates the Lemonade SDK with Muse-Glimmer 30B and an experimental ROCm image-generation module. More than a benchmark event, this is a signal for local LLM adopters: the value lies in CPU, GPU, and NPU optimization, cost predictability, and data...
AMD releases Lemonade SDK 11.6, integrating the 30-billion-parameter Muse-Glimmer LLM and an experimental ROCm-based image generation module called TheNoise. The update strengthens the open-source stack for running local AI apps on CPUs, GPUs, and NP...
AMD and Meta have upstreamed FP8 optimizations for Instinct GPUs into PyTorch: a 13.4% throughput gain on Llama3-8B, 89% recovery of quantization overhead on DeepSeek-V3 671B, and up to 6.2x faster MoE kernels. Teams can now get competitive FP8 train...
A recurrent retrofit on Qwen2.5-0.5B-Instruct enables iterative latent reasoning at small parameter budgets. It matches the full block, extrapolates supervised depth, and answers 7.6 times faster than a scratchpad model. The inverse task reveals cata...
Intel's LLM-Scaler project adds same-day support for Muse Glimmer, strengthening a Docker-based stack for Arc B-Series GPUs that includes vLLM, SGLang, and ComfyUI. This move makes on-premise inference more accessible and signals Intel's commitment t...
Unsloth Desktop is the first open-source app for running and training LLMs locally. It spans Windows, macOS, Linux and supports NVIDIA, AMD, Intel, and Mac hardware. It claims 2× faster training, 70% less VRAM, private search, RAG, MCP, and an OpenAI...
FastFlowLM 1.0, the open-source software for running language and multimodal models on Ryzen AI NPUs, officially joins the ROCm ecosystem. The move signals AMD’s intent to deliver a unified stack for local inference, from discrete GPUs to integrated ...
An experiment shows that for $200 on rented GPUs you can train a 1.1B parameter LLM and deploy it on CPU or even a smartwatch. The negligible cost rewrites the TCO calculus for organizations evaluating self-hosted solutions: data sovereignty becomes ...
A personal project demonstrates how to train a 1.1-billion-parameter LLM from scratch on 20 billion tokens for around $200. The model, fine-tuned with LoRA and quantized, runs on CPU and even on a WearOS watch. Concrete evidence that data sovereignty...
A new approach called TEXAS exploits how mixture-of-experts models activate their sub-models to steer fine-tuning, focusing supervision only on relevant tokens. For those running LLMs on-premises, this means more efficient adaptation on proprietary d...
Pathway introduces BDH, a post-transformer architecture matching GPT-2 scaling from 10 million to 1 billion parameters while training on ordinary GPUs. A signal for those seeking self-hosted LLMs without extreme hardware constraints.
The official INT4 of Ling-3.0-flash runs on a single DGX Spark, but the default configuration delivers only 20.8 tok/s. Two changes — dropping `--enforce-eager` to enable CUDA graphs and switching on MTP speculative decode with one token — push it to...
