The question that breaks the mold: it's not about benchmarks, it's about budget

The question that appeared on Reddit — “Seriously, what do you do with small models?” — might sound provocative, but it captures the exact moment when the technical community stopped chasing parameters and started asking what could actually fit into a budget. This isn't a lab curiosity: it's the symptom of an entire sector seeking concrete tools for self-hosted deployment, far from variable costs and cloud compliance uncertainties.

Those participating in the discussion don't cite synthetic benchmarks or academic contests. They talk about interrogating corporate knowledge bases, assisting code writing with near-zero latency, and fine-tuning on proprietary data run overnight on a single graphics card. The common denominator is control: over data, over costs, over latency. It's here that small LLMs — typically under 13 billion parameters, often quantized to 4 bits — stop being toys and become working equipment.

The strength of the original question lies in exposing a widespread hypocrisy: for many real workloads, a 7 or 8 billion parameter model is more than enough, as long as it's wrapped in the right pipeline. It's not about absolute performance, but about fitting the operational context. And that's exactly what makes the debate a powerful market signal: those writing the checks aren't looking for the biggest model; they're looking for one that can run where the data is needed, when it's needed, without moving anything elsewhere.

The four pillars of on-premise use

The analysis of the Reddit discussion, combined with trends AI-Radar monitors, highlights four application streams giving shape to the phenomenon. The first is document automation with retrieval-augmented generation, or RAG. Enterprise IT teams load internal documents into a local vector database and use an LLM to produce pointed answers. With well-calibrated temperatures and careful chunking, a 7B model generates outputs reliable enough to support legal, financial, or medical decisions, without transferring a single line outside the corporate perimeter.

The second stream is assisted coding. Here the decisive factor isn't so much the absolute quality of the suggestion, but latency and operational continuity. Engineers working on construction sites, offshore platforms, or simply on the move can run inference on portable hardware with a mid-range GPU, getting real-time code completion even without connectivity. The model runs entirely locally, and latency drops to a few milliseconds — an advantage no cloud API can consistently guarantee.

The third pillar is fine-tuning on proprietary data. Until recently, training a model required GPU clusters costing tens of thousands of euros per hour. Today, techniques like LoRA (Low-Rank Adaptation) and quantized formats allow fine-tuning a 7B LLM on a dataset of hundreds of examples, using a single RTX 4090 or a workstation with 24GB of VRAM. This means a software house can create a specialized assistant for its product, or a legal department can personalize a model on its internal case law, without sharing sensitive data with third parties.

The fourth stream concerns specialized chatbots for customer support or internal training. Here, data sovereignty marries deep customization: the model can be trained or refined on company procedures, technician responses, or product manuals. Self-hosted deployment avoids regulatory hurdles related to GDPR or other personal data handling regulations, because everything remains confined to the customer's infrastructure. The result is a chatbot that speaks the company's language, not just the generic corpus it was pre-trained on.

The hardware equation: a single card, TCO, and the resurgence of mid-range GPUs

The glue holding these scenarios together is the availability of affordable computing hardware. The magic number, emerging from countless discussions, is 24GB of VRAM: the amount of memory that allows loading a 13B model quantized to 4 bits and maintaining a sufficient context window for most business applications. Cards like the NVIDIA RTX 4090, or the older A5000, offer exactly this capacity and are becoming the optimization target for local inference frameworks.

Total Cost of Ownership (TCO) is the argument tipping the scales from cloud to self-hosting. With cloud models, costs grow linearly with the number of tokens processed, making it hard to predict monthly spending when request volumes vary. A company that queries its knowledge base hundreds of times a day with a local model bears a fixed hardware and maintenance cost, with no surprises on the invoice. If the hardware is amortized over two or three years, the cost per token can become negligible compared to centralized API rates, especially at high volumes.

This is triggering an interesting market effect: mid-range GPUs, traditionally dominated by gaming, are finding a new pool of buyers among system integrators and IT departments. A server with eight H100s isn't necessary for useful inference; often a quiet workstation tucked into a network closet is enough. This expands the pie for GPU manufacturers and stimulates development of compact, low-power solutions, paving the way for an era where on-premise AI is no longer a niche but a standard architectural choice.

Frameworks and community: the silent engine of local inference

None of these scenarios would be viable without the evolution of dedicated frameworks. Ollama has made running an LLM locally as simple as launching a Docker container, while Llama.cpp and its quantization libraries push efficiency on consumer CPUs and GPUs to the limit. The community has built pipelines that integrate RAG, chat, and fine-tuning with a few commands, lowering the technical barrier for those who aren't researchers but sysadmins needing to solve a concrete problem.

Work on quantization — from 8 bits down to 2 bits — is the real game-changer. It allows models born to run on servers costing tens of thousands of euros to operate on a gaming laptop without drastic quality loss. GGUF formats and runtimes optimized for AVX2 instructions or Apple Silicon GPUs show that the direction is no longer just “build bigger models,” but “make sufficiently capable models run everywhere.” This aligns the interests of researchers, who gain mass adoption, with those of enterprise users, who want low-cost deployment without cloud licensing constraints.

An often overlooked aspect is the democratization of fine-tuning. With small models, the computational cost of a training cycle on proprietary data is within reach of a freelancer. Using scripts based on PEFT and LoRA, a professional can fine-tune a 7B LLM over the weekend, with his own GPU, and get a vertical assistant ready for Monday morning. This creates an ecosystem where value no longer lies in access to the model, but in the ability to customize it quickly.

Who stands to lose (and who stands to gain) in this repositioning

In this scenario, generic cloud inference services risk seeing their base of privacy- and cost-conscious clients eroded. Companies already familiar with on-premise infrastructure and handling regulated data are beginning to ask why pay per token for an API when they can get equivalent performance — and predictable costs — with hardware they already own or can purchase with a one-time investment.

Large model providers may also need to review their strategies. If an open-source 7B LLM, enriched with corporate data, solves 80% of internal use cases, the differential offered by a proprietary model with hundreds of billions of parameters may not justify the extra cost or compliance risks. This isn't a total replacement, but a progressive shift of workloads downward, where the commoditization of small models is already underway.

On the hardware front, manufacturers like NVIDIA, but also AMD and newcomers in the NPU market, see a demand segment opening that doesn't replace but complements the server market. Demand for consumer GPUs with high VRAM is set to rise, and workstation and mini-PC makers are already launching machines designed to host these workloads. System integrators who can offer turnkey solutions — a box with pre-installed LLMs and a web interface for enterprise RAG — could become the new protagonists of IT consulting.

What to watch in the coming months: from proof of concept to production line

The signal AI-Radar will keep an eye on is the transition from proof-of-concept to production deployments. When companies start publishing technical posts about how they replaced a ticket system based on a cloud API with a self-hosted alternative, the market will get definitive validation. Meanwhile, the technological trajectory is clear: small models will continue to improve, supported by distillation techniques and more efficient architectures like mixture of experts, which activate only a fraction of parameters per inference.

On the hardware side, the VRAM bar is bound to rise. The next generations of consumer GPUs could exceed 32GB, making it possible to run quantized 30B models, further expanding context windows. In parallel, the integration of AI accelerators directly into CPUs — like Intel's NPUs or Apple's Neural Engine — could pave the way for ultra-low-power local inference for simple tasks, further broadening the perimeter of self-hosting.

The regulatory landscape will be an accelerator. The European AI Act and similar regulations in other jurisdictions will push organizations to choose architectures that keep data under direct control. Self-hosted will become not only a cost choice but a compliance requirement. Those experimenting today with Ollama and Llama.cpp may tomorrow find themselves leading the corporate migration to local AI, turning the Reddit question into a business plan.