The trend towards running AI models locally on devices, from laptops to edge hardware, is accelerating. This involves developing compact models, efficient architectures, and specialized hardware (like NPUs in weights) to enable faster, more private, and cost-effective inference.
A local 27-billion-parameter model detected Blender through MCP and wrote Python code to speed up a calculation without an explicit request. The episode shifts focus from VRAM and throughput to tool-access governance: sandboxing, permissions, and pol...
A user reports that a locally running Qwen 3.8 27B model discovered access to Blender via MCP and wrote Python code to speed up math calculations. The episode shows unexpected autonomy in self-hosted models, raising governance and security questions ...
ISTA DASLab has released a Coder build of DeepSeek-V4.1-Flash-1017 with half its routed experts removed and transformer weights compressed to 1 bit per parameter. The GGUF takes 72.1 GB of transformer weights, down from 138.2 GB unpruned, and retains...
A 9-billion-parameter preview on Hugging Face highlights how vendors use early releases to test demand for self-hosted LLMs. With no benchmarks or hardware details, the value lies in validating pipelines and VRAM usage before on-premise decisions.
A LocalLLaMA user proposes moving beyond token/s chasing and building private quality benchmarks, especially across quantization levels. The idea: keep test data private and share only comparative charts. It is a reaction to benchmark noise and score...
The JetBrains/Mellum2.1-12B-A2.5B-Thinking-MTP repository on Hugging Face introduces an integrated multi-token prediction head that accelerates decoding by 1.4–1.8x while accepting about 90% of proposed tokens. vLLM 0.33, transformers 5.20 and llama....
A researcher compiled an NPU into Qwen 3.5-4B weights and ran stories260k without external runtimes. QwenPU redefines the boundary between model and platform: weights become a computational substrate, with implications for self-hosted deployment, aud...
QwenPU compiles an NPU into Qwen 3.5-4B's weights to run Karpathy's stories260k, with no external tools and bit-exact operations. The project redraws the boundary between model and compute platform, with implications for self-hosted stacks and data s...
Microsoft shows that a 4-billion-parameter vision-language model can hit 66% on OSWorld-Verified, using 13% fewer model calls and 25.4% fewer output tokens after online RL. The secret is a five-stage training pipeline and workflow-first synthetic env...
llama.cpp v0.8.0 adds compatibility with DFlash2 for Qwen3.8-Flash-Next. Tests on Apple Silicon M5 Max with quantized GGUFs show an average 17% decoding throughput gain over MTP, with peaks of 39% on list-like prompts and a minor prose regression. Th...
SenseNova brings to Hugging Face a multimodal LLM with reused decoder blocks and hybrid attention. With 24 layers run twice and 56 executions from 32 distinct layers, it targets agentic workloads and documents up to 256K tokens. The approach reduces ...
Liquid AI has introduced LFM3, a family of compact models from 1.7–2.2 billion parameters with a tri-hybrid architecture and llama.cpp support. The decision models skip token generation and return calibrated answers in a single forward pass. For on-p...
An open System One model runs locally on a laptop and reaches results close to GPT-4 on medical benchmarks, with well-controlled calibration on knowledge tests but less reliable patient-level predictions. A two-parameter recalibration method using 50...
Liquid AI has released a family of models ranging from 1.7 to 2.2 billion parameters, designed to run on phones, laptops, and edge devices. Beyond text, there are multimodal versions and decision models that return answers in a single pass. The commo...
inclusionAI updates Ling-3.1-flash: 560B total parameters, 25B active per token, and a 1M token context window. Agentic and scientific benchmarks improve. The base checkpoint is opened for research. The article analyzes implications for self-hosting ...
