Driven by cost, privacy, and control, organizations are increasingly deploying small language models locally. Innovations in quantization, efficient architectures, and specialized hardware are making on-premise AI viable for a wide range of use cases.
thinkingmachines has released Inkling-Small, a mixture-of-experts LLM with 276 billion parameters, only 12 billion active, a 1 million token context window, and NVFP4 and GGUF quantizations. AI-RADAR analysis focuses on the hidden challenge of the KV...
A paper evaluates recursive transformer architectures for thermo-mechanical simulations in semiconductors. Result: depth-wise weight sharing reduces overfitting and FLOPs, a direct advantage for those running models on limited hardware while keeping ...
A user buys an RTX 5090 to run local models and escape cloud API fees. Upgrades to two RTX 6000 Pro cards for 100B models, only to find that most daily tasks worked fine on the single 5090. A lesson on hype, hardware, and the real needs of AI self-ho...
A user experiment runs the Kimi K3 model locally with Q2_K quantization, 768GB of system RAM and two NVIDIA 5090 GPUs. Prefill at 50-70 tokens/s and decoding around 4 tokens/s, but the generation speed increases over time, hinting at cache or swap dy...
Moonshot releases Kimi K3, a 2.8T-parameter Mixture-of-Experts model quantized in MXFP4. On-premise deployment shows that an 8×A100 node (640 GB) needs three machines before KV cache allocation; 8×H200 (1.13 TB) requires two nodes. Only 8×B300 (2.3 T...
The Spanish scaleup raises $570M at a $1.7B valuation, betting on CompactifAI technology that shrinks LLM size by up to 95% with negligible accuracy loss. The goal: move inference from data centers to edge devices, reshaping costs, energy consumption...
The Reddit question “What do you actually do with small models?” reveals a reshaping of AI infrastructure far from data centers. AI-Radar analyzes four real-world use cases, the crucial role of VRAM, TCO, and local frameworks, and how data sovereignt...
A new study strikes a middle ground between lossy and lossless compression, achieving quantization that preserves statistical output quality while accelerating inference and cutting bits per parameter down to 3.3. Direct implications for anyone runni...
Swiss-AI releases Apertus 1.5, a family of 8B and 70B parameter LLMs that is fully open: weights, training data, and procedures are public. Multimodal (images, audio, text), up to 262,144 token context, and improved reasoning mode. Its maximally tran...
