The number that jumps out is not the solve rate, but token consumption: 3.7 million versus 10 million per run. In a benchmark run by a user on 14 cross-system tasks with three MCP servers behind them — a CRM, an issue tracker, and a doc store — Claude Managed Agents with Opus 4.8 solved 11 of 14 tasks at $11.8 per run and 10 million tokens. TrueForge, an open-source runtime, solved the same number of tasks with the same Opus model, but at $8.6 per run and 3.7 million tokens: 63% fewer tokens and 30% lower cost. The average tool calls per task also dropped from 32 to 19.

This difference is not only a cloud billing issue. In agent workloads, token consumption determines how many concurrent agents can run on the same inference capacity, how much VRAM remains available for context, and how much headroom is needed before hitting request limits. The user who ran the test, after also trying Claude Code, DeepAgents, OpenCode, and Pi, considers Claude Code the most mature for the managed experience, but points to high costs and token burn. DeepAgents offers a middle ground for teams looking for a more structured open-source agent framework.

Switching the model on TrueForge changes the economics further: with GLM-5.2, the benchmark shows an average of 11.7 solved tasks out of 14, $3.0 per run, and 3.8 million tokens. That is a slightly higher average solve rate than Claude Managed Agents with Opus at roughly 75% lower cost. The point is not to pick a vendor, but to observe that in an open stack the model choice can be separated from the runtime: TrueForge keeps the orchestration stable while allowing model swaps to align cost, latency, or data residency constraints.

The trade-off is operational maturity. The same user notes that the open-source runtime still lacks first-class tracing and evaluation tooling, does not ship its own code-execution sandbox, and uses intentionally lossy context compaction. Anyone deploying it on their own infrastructure must integrate those pieces and take on security work that a managed platform includes in its price. Token reduction is therefore not an automatic saving; it becomes part of TCO only after the control layer is built.

This is where the story moves from a harness ranking to a structural signal. In self-hosted deployments, where the marginal cost is not the token but the GPU hour and energy, a runtime that cuts tokens per task by 63% increases useful work per unit of capacity. Model neutrality and the ability to run on your own infrastructure shift the incentive from vendor lock-in to orchestration design. The benchmark kit is public in the TrueForge repository. For teams evaluating these scenarios, AI-RADAR offers frameworks on /llm-onpremise to compare trade-offs between managed harnesses and self-hosted runtimes.