Shopify has described a case study that changes where production experience accumulates. Instead of only improving the prompts around an LLM, the company uses real failures to update the model weights. The clearest example is the GraphQL agent, which answers merchant questions by writing and running queries against the Admin GraphQL API. In production it handles up to 2,000 requests per minute. The first figure is economic: serving that traffic with a frontier model would cost an estimated $27 million per year, while the specialized model comes in closer to $1 million. That is a 96% reduction in serving cost, with quality surpassing the frontier baseline.

The starting point is not the model but the definition of quality. Shopify builds a rubric that turns product requirements into scored criteria: completeness, execution, response quality, safety. The ground truth includes not only curated examples but randomly sampled traffic. A judge is calibrated with DSPy and reflective optimizers such as GEPA and ACE. Before that, two expert annotators blindly label 25 random samples and the team measures agreement with Cohen's kappa. If it is too low, the rubric is ambiguous and must be iterated. The reason is straightforward: if it confuses product experts who work on the product every day, it will confuse an LLM too.

From harness to weights

Frontier models have frozen weights. Corrections pile up outside the model: in prompts, retrieval examples, routing rules, and code. Shopify first pushes that harness as far as it will go, treating it as an autoresearch problem. An agent proposes changes to prompts, tool definitions, or harness code, evaluates them against the judge, and keeps only improvements. When those gains plateau, the loop moves into parameter space. The system mines anonymized production traffic for hard cases the judge scores poorly. A panel of frontier reasoning models corrects them; an arbiter merges the critiques into a repair instruction; the conversation is replayed and scored again. If it passes, it becomes a trajectory for reinforcement learning. If not, it goes to human annotation, with Toloka applying the same rubric used to calibrate the judge.

Training happens in two stages. The first is supervised fine-tuning on full trajectories, including reasoning, not just final answers. The second is GRPO, using the calibrated judge as reward signal. The self-healing pipeline runs daily and adds new trajectories to the training corpus. Shopify runs full-parameter fine-tuning over the accumulated data and then repeats GRPO. PyTorch distributes training across GPUs with tensor, context, and data parallelism.

The inference bill

A better model still has to run. The agent's system prompt is long and static, and every generated token has to attend over the whole prefix. Gist compression reduces the prompt from roughly 6,000 to about 1,500 learned tokens, with no measured quality loss on the judge. In a load test at 350 requests per minute, time-to-first-token drops about 19% and end-to-end latency drops about 38%. The same compression increases throughput by about 16% in requests per second and about 12% in output tokens per second on identical GPUs, equivalent to roughly 14% fewer GPUs for the same traffic.

This is the most relevant structural signal. The Shopify case does not simply show that a small model can beat a large one on a specific task. It shows that serving cost stops being tied to a per-token price from an external API and becomes an infrastructure TCO problem again. vLLM, an inference engine built on PyTorch, keeps throughput high with continuous batching and copes well with tool-call heavy workloads. For teams evaluating on-premise deployment, the compression and GPU numbers are the reference point: it is not about replicating a frontier model, but about building the loop that turns traffic into training.

The prerequisite is volume. You need enough real conversations to produce hard negatives and a reliable judge. A team with few users does not have the same raw material. But when the traffic is there, the advantage is not in a more elaborate prompt: it is in owning the learning loop. For teams evaluating on-premise deployment, AI-RADAR offers analytical frameworks at /llm-onpremise to compare these trade-offs. That advantage is much harder to copy.