A community-driven shift toward smaller LLMs and quantization techniques is enabling efficient local inference, lowering costs, and preserving data sovereignty. Tools like llama.cpp, statistically lossless quantization, and compact model architectures are at the forefront.
The Spanish scaleup raises $570M at a $1.7B valuation, betting on CompactifAI technology that shrinks LLM size by up to 95% with negligible accuracy loss. The goal: move inference from data centers to edge devices, reshaping costs, energy consumption...
The Reddit question “What do you actually do with small models?” reveals a reshaping of AI infrastructure far from data centers. AI-Radar analyzes four real-world use cases, the crucial role of VRAM, TCO, and local frameworks, and how data sovereignt...
A simple Reddit question sparks a practical debate: who's using small language models, and for what? The thread reveals a maturing ecosystem where computational efficiency and data sovereignty matter more than headline-grabbing parameter counts.
A new paper introduces statistically-lossless quantization (SLQ), offering probabilistic guarantees on output quality without sacrificing speed. It defines three losslessness levels: task-lossless under 4 bits, distribution-lossless (output distribut...
A new study strikes a middle ground between lossy and lossless compression, achieving quantization that preserves statistical output quality while accelerating inference and cutting bits per parameter down to 3.3. Direct implications for anyone runni...
The leading open-source framework for LLM inference on consumer CPUs and GPUs now supports Laguna XS.2 and M.1 cards, accelerators built for on-prem AI workloads. The direct integration in commit b10087 signals a maturing specialized hardware ecosyst...
Upstage releases Solar-Open2-250B, an open-weight model purpose-built for agentic workflows, with a hybrid MoE architecture: 250B total parameters but only 15B active per token. Linear attention and removal of positional encoding enable a 1-million-t...
