Prompt caching is one of those silent optimizations that make large-scale LLM inference economically viable. When it works, the system reuses computations already performed for the parts of the prompt that remain identical across requests, avoiding recomputation of the key-value cache from scratch. The gains in throughput and latency are immediate, and the computational cost drops accordingly—a no-brainer for the cloud, and vital for on-prem deployments where every GPU hour counts toward the Total Cost of Ownership.

But what happens when a seemingly harmless formatting detail wrecks the entire mechanism? That’s exactly what user CharlesStross discovered while poking at DeepSeek-V4-Flash-0731, and the Reddit post they published is a gem of practical reverse engineering.

The crux is the chat template. DeepSeek doesn’t ship a standard jinja, but any distribution that faithfully reconstructs what the company releases in its python chat template ends up with a peculiar behavior: every system message, no matter where it appears in the conversation, gets hoisted to the top of the prompt, as if it were an initial preamble. The format simply doesn’t allow mid-conversation system turns. So, inserting a system role anywhere other than the header dirties the entire prefix: prompt caching gets invalidated because the supposedly immutable initial portion changes every time you add or modify a system message in the middle of the dialogue.

The damage is far from trivial. For users of hosted services, it means API calls that consume more tokens than necessary and cost more; for those running inference locally with llama.cpp or similar engines, the blame for “garbage” prompt caching falls on increased end-to-end latency and GPU resource consumption that could have been avoided. The author admits to wasting time and frustration before pinpointing the culprit, and it’s precisely to spare others the same ordeal that they shared the fix.

The latest_reminder role: how it should have been done from the start

There is a way out, and it’s called latest_reminder. It’s a role that DeepSeek specifically trained to handle mid-conversation disambiguations, for those very cases where many templates would use a system message in the middle. In essence, latest_reminder does the job that a developer’s instinct would assign to a system message inserted later on: it conveys an instruction or a constraint that must be effective from a certain point onward, without disturbing the prefix. The engine must support it, of course; llama.cpp does so smoothly, while on other runtimes compatibility needs to be checked.

This apparently technical detail signals a broader tension living throughout the LLM ecosystem. On one hand, the flexibility of chat templates allows each lab to shape the conversation format according to its own training and alignment choices. On the other, implementations running on third-party serving engines (vLLM, TGI, Ollama, llama.cpp) must faithfully reconstruct that format, and any deviation can generate sneaky bottlenecks that don’t surface during simple correctness tests but only under real load, when the cache stops working.

In this case, the uncertainty lies in the absence of an explicit format contract: if DeepSeek doesn’t provide an official jinja, anyone distributing quantized versions or deploying the model must reverse-engineer DeepSeek’s python code. An anomaly that developers working in on-prem environments—where control over the serving stack is total but so is the responsibility for every optimization—know all too well.

Broken cache, busted budget

The takeaway for DeepSeek-V4-Flash users is clear: never put a system message far from its designated spot, the initial one. The alternatives are two: either use latest_reminder for post-prefix messages, or accept losing the caching advantage, with the attendant consequences in cost and user experience.

For those evaluating on-prem LLM deployments, episodes like this are a reminder that real efficiency isn’t measured only in model parameters or peak throughput on benchmarks, but in the ability to squeeze every drop from computational savings. A template that ignores the semantics of caching can devour resources believed to be safe, and a small design detail—using a dedicated role instead of abusing system—makes the difference between a service that scales economically and one that burns GPU budget on repeated calculations. CharlesStross’s advice, in its dry power-user directness, is the kind of thing that should be printed and pinned on every MLOps engineer’s wall.