For decades, the memory market was the most boring corner of the semiconductor industry. DRAM and NAND flash behaved like any commodity: prices fell as manufacturing processes matured, interrupted only by predictable boom-and-bust cycles. That rule no longer holds. The AI race has created an unprecedented distortion, and the bill could be steep.

The reason: LLM workloads and model training are bandwidth-hungry. The most powerful GPUs rely on high-bandwidth memory (HBM), vertically stacked DRAM dies that deliver extreme throughput at the cost of complex manufacturing. Demand for HBM has exploded, pushing producers to allocate ever-larger shares of their capacity to a segment that was niche just a few years ago. The result: prices are soaring instead of declining.

The bottleneck won't ease quickly. Building new HBM production lines requires billions in investment and years of work; estimates point to relief no sooner than 2028. In the meantime, major cloud providers are hoovering up most of the supply, leaving little for the rest of the market.

For organizations evaluating on-premise AI infrastructure, this has direct consequences. The cost of nodes equipped with cutting-edge GPUs swells, making it harder to justify Total Cost of Ownership (TCO) compared to the cloud for intermittent or experimental workloads. Entities that must keep inference on-site for data sovereignty or latency reasons face longer lead times and higher budgets. Some are already reconsidering quantization and smaller models that can run on less memory-demanding hardware.

There is a second, more structural effect. The current investment cycle is widening a dangerous gap between production capacity for AI memory and that for standard DRAM. If HBM demand slows — for example because generative AI adoption fails to meet growth expectations, or because new architectures reduce bandwidth requirements — manufacturers could be stuck with expensive excess capacity. The memory market, notorious for violent downturns, might see its next bust amplified by this concentration of bets on a single segment.

The memory market anomaly thus mirrors the speculative phase the AI industry is living through. For those designing on-premise infrastructure, the signal is clear: capacity planning must account for price volatility unseen in recent history, and architectural flexibility — from quantization levels to the ability to shift some workloads to the cloud — will become a competitive differentiator.