Anyone running LLMs locally knows the frustration: a small tweak to the system prompt, a change in message order, or even a parameter like reasoning_effort suddenly turns a fast response into a slow GPU recomputation. The prefill cost, already noticeable on consumer hardware, becomes a hidden tax on self-hosted efficiency.

Cache-hunter, a tool built by a community maker, tackles the problem at its source. It works as a proxy between your harness and the LLM endpoint: it captures sessions, tracks hashes of repeated components, and displays a table where any break in stability shows up as a red cell. The author tested it with their own harness and with several well-known clients – including Claude Code, Cline, Pi, Hermes, and Vibe – and found unstable system prompts, unexpected tool variations, or changed message order in most cases.

The design is minimal but hits a sore spot: few harnesses systematically check the reproducibility of their calls. When building a conversational interface or an agent, the focus tends to be on response correctness, while inference engines optimize latency via KV-caching. A solid cache can cut time-to-first-token by up to 70%, but any structural divergence invalidates it. On a GPU cluster the overhead is manageable; on a single card in a cabinet or on a desk, the difference is tangible – on the energy bill too.

Why caching is a maturity signal for harnesses

The emergence of a tool like cache-hunter marks a specific moment in the evolution of local stacks: we are moving from the thrill of “it works” to a demand for predictability and cost containment. Harness builders are no longer just hobbyists; self-hosted LLMs increasingly serve small businesses, law firms, and organizations handling regulated data. For them, inference efficiency is not a luxury but a factor in Total Cost of Ownership, alongside data sovereignty.

Cache-hunter introduces no new techniques – change detection via hashing is a well-known pattern – but it makes that diagnostic accessible inside a development workflow. The most interesting side effect is that it forces you to think about how each call is composed: the data structure for messages, whether the system prompt gets rebuilt every time, whether tools preserve consistent ordering and content. A harness that fails the cache-hunter test is, in effect, requesting more computation than necessary with every interaction.

The direction is clear: on-premise LLM infrastructure is absorbing observability tools that previously belonged only to cloud distributed systems. It signals that the DIY community is not only maturing to the level of commercial products but in some cases getting there first, driven by a more direct relationship with hardware and its physical limits. Cache-hunter is a brick in that infrastructure: small, specialized, solving a real problem without adding complexity. And it reminds us that the cost of a model isn’t measured only in tokens per second, but in how many of those tokens we could have avoided.