When a Reddit user recounted buying an NVIDIA RTX 5090 to run 27-billion-parameter models locally, the goal was clear: break free from cloud APIs and recurring costs. They started with LoRA fine-tuning on their own data, some RAG pipelines, and a 130k token context window compressed with Q8 quantization. The results were satisfying, but the video memory was already straining.

So in came two RTX 6000 Pro cards, each with 48 GB, for a total of 96 GB VRAM. Meanwhile, 100-billion-plus parameter models started dropping everywhere, making even that setup seem insufficient. The idea of a 512 GB cluster felt like the only path forward.

Then came the turning point: most of the real-world tasks—the very reason they’d bought that first GPU—ran perfectly on a single 5090. Everything else was idle compute, so much so that they started lending the excess cards to friends.

This personal tale mirrors a tension running through the entire on-premise AI ecosystem. On one hand, the race toward ever-larger models—fueled by labs and marketing—breeds a sense of inadequacy among those managing local infrastructure. On the other, the daily reality of developers, researchers, and small businesses shows that models in the 7-30 billion parameter range, optimized with quantization and extended context, cover the overwhelming majority of use cases.

Hardware costs tell a similar story. A top-tier consumer GPU costs a couple thousand euros, while two professional cards easily surpass €10,000. When you factor in power, cooling, and physical space, the mini datacenter becomes an investment that needs sustained workloads to break even. Yet over-provisioning is a constant temptation: the fear of falling behind the latest open-weight release drives people to buy more VRAM than they actually need.

The choice to lend out excess compute to friends points to an interesting dynamic: a micro-sharing economy for resources that replicates, on a small scale and with full data control, the logic of cloud services. It’s no coincidence that the second-hand AI GPU market is booming and peer-to-peer lending platforms are beginning to appear.

For those considering building their own on-premise stack, defining real computational needs is the most critical—and often most overlooked—step. The temptation to over-dimension for fear of being left behind can turn an economically viable project into a disproportionate investment. AI-RADAR offers analytical frameworks to map these trade-offs, helping to size infrastructure against actual requirements rather than hype.

Next time someone tells you that you need hundreds of gigabytes of VRAM for local inference, maybe it’s worth asking: what am I really going to do with it? The answer could be far more modest—and far smarter—than expected.