For a local LLM agent, active context is a scarce resource. Memory capacity, prefill latency, cache growth, and service objectives all limit how many input tokens each call can afford. BudgetBench starts from that constraint but shifts the frame: instead of measuring memory strategies at a single context length, it treats the per-call token budget as the independent variable. The protocol holds model, task, sampler, and decoding fixed while sweeping budgets across 2K, 4K, 8K, 16K, and 32K tokens, recording quality, budget utilization, latency, and budget-violation rates.

The core contribution is not a final ranking. It is a reusable measurement surface: a swappable MemoryStrategy contract, explicit budget enforcement, deterministic or versioned graders, prompt-audit metadata, and reproducibility artifacts. That is not an implementation detail. In a local deployment, budget is not an academic parameter; it is a service constraint touching VRAM, latency, and per-call cost.

The pilots do not produce definitive rankings. Across a local qwen2.5:1.5b pilot with 89 items each on SWE-bench Verified and LongBench v2, a hosted 50-item Qwen3 30B-A3B replication with exact tokenization, and a 500-item LongMemEval oracle study scored by the official GPT-4o evaluator, the harness exposes budget-compliance failures, non-monotonic quality curves, and operating points that single-budget evaluation hides.

The most important result may be what the source leaves unresolved: the comparison between reduced-budget strategies and full context has no clear direction. In the local pilot the effect is near null, while the hosted replication favors full context in the point estimate. That should not be read as a failure. It suggests that the choice of memory strategy cannot be separated from the deployment stack: tokenization, model, hardware, and latency all shape the operating point.

For teams evaluating on-premise stacks, there is a structural signal. Budget constraints turn evaluation from “how good is the model” into “how disciplined is the agent in respecting a service contract.” An agent that produces high-quality answers but repeatedly exceeds budget can degrade tail latency, inflate cache growth, and shift TCO in nonlinear ways. Treating violation rates as a first-class outcome, rather than a secondary metric, is meaningful.

Transparency is part of the protocol. The early pilot used a tokenizer approximation that undercounts some served-model prompts; its violation rows are diagnostics of that approximation, not claim-bearing results. Timings are operational diagnostics as well. This failure-reporting discipline may be the most useful legacy for teams that need to scale fixed-budget memory-strategy evaluation. For those weighing on-premise deployment decisions, AI-RADAR offers analytical frameworks for specific trade-offs, but BudgetBench clarifies that the variable to isolate is budget, not just quality.