“Seriously, what do you do with them?” The question, recently posted on a Reddit forum, might sound trivial, but it touches the exposed nerve of a whole generation of developers and system integrators betting on lightweight LLMs. This isn't an academic query: shifting focus from lab benchmarks to daily workloads reveals a far more compelling — and often more profitable — picture than the one painted by trillion-parameter races.

The thread highlights a structural trend: compact models (typically under 13 billion parameters, often quantized to 4 bits) are carving out a specific niche outside big tech data centers. It's no surprise that self-hosted deployment is the common denominator. In environments where data cannot leave the corporate perimeter — law firms, hospitals, finance departments — a model that runs entirely on a single card with 24 GB of VRAM becomes an enabler, not a compromise. The trade-off with cloud alternatives isn't merely technical: it's an architectural choice that flips the “pay-per-token” logic in favor of predictable TCO.

From an application standpoint, four directions emerge. First, document automation with retrieval-augmented generation (RAG): 7B or 8B parameter models, properly tuned, are enough to query internal knowledge bases without unacceptable hallucination. Second, assisted coding, where ultra-low inference latency on local machines ensures business continuity even without connectivity. Third, perhaps the most promising, fine-tuning on proprietary data: at these model sizes the computational cost remains manageable even on high-end consumer hardware, democratizing a process that until yesterday required H100 clusters. Fourth, specialized chatbots for customer support or internal training, where full control over data flow avoids GDPR headaches when handling sensitive information.

Who loses in this reshuffle? Generic cloud inference APIs may see their enterprise privacy-conscious client base erode, while manufacturers of mid-range GPUs (RTX 4090, future generations with 32 GB or more) find a market that's no longer just gaming. It's no coincidence that the most active frameworks, from Ollama to Llama.cpp, keep optimizing precisely for the “small models on local hardware” segment. The structural message is clear: the scale race isn't the only game in town. The ability to extract value from models that run where they are needed, when they are needed, with data never transferred elsewhere, is becoming the real benchmark for those signing checks. For anyone evaluating on-premise deployment today, the question is no longer whether a small LLM can serve, but how to choose among the many arriving on the market.

The community knows it: the Reddit discussion isn't looking for models to show off, but tools to budget for. And that changes everything.