The leaked audio that is embarrassing top-tier consulting firms is a whisper turned roar: nobody really knows how to measure the effectiveness of generative AI. As IT departments push more and more tokens — the so-called "tokenmaxxing" — the bill climbs without any commensurate measure of value delivered.

The recording, obtained from an anonymous source, captures senior partners of a major advisory firm admitting that clients demand concrete KPIs, but traditional evaluation models fail to keep up with the fluidity of Large Language Models. The paradox is glaring: companies spend to get longer texts, summaries, and generated code, but "success" remains a fuzzy concept, often gauged by rough estimates or individual productivity metrics that are hard to isolate.

The tokenmaxxing paradox

Behind the crypto-mining slang borrowed by AI — "tokenmaxxing" — lies an increasingly common practice: sending huge prompts, requesting extensive outputs, chaining multiple LLM calls in pursuit of deeper insights or sophisticated automations. Every extra token, however, carries a computational cost that translates into a larger cloud bill or, for those running on-premise hardware, into saturated VRAM and inference queues.

The trend is amplified by adoption of ever-larger models with context windows exceeding a million tokens. Whoever foots the bill — typically the CFO — begins to wonder whether the company is buying productivity or just digital chatter sold by the pound.

Measuring the unmeasurable

The issue raised by the leak is not technical, but methodological. Classic metrics — cost per token, latency, throughput — remain useful efficiency indicators, yet they say nothing about business effectiveness. What is the value of a slightly more accurate answer? How do you monetize a 15% reduction in an analyst's time? Consulting firms themselves struggle to build measurement frameworks that are more than qualitative guesswork.

For organizations considering on-premise deployments, this gap is even more critical. Those who invest in GPU servers, local storage, and self-hosted inference stacks often do so to control Total Cost of Ownership and data sovereignty. But without outcome metrics, TCO risks becoming a mere accounting exercise with no bearing on real business impact.

Trade-offs and instruments to avoid sailing blind

At stake is a classic industry trade-off: the urge to experiment without constraints versus the pressure to justify every euro spent. Choosing on-prem can eliminate variable cloud token costs, but it introduces upfront capital expenditure and the need to size hardware for peak usage, with no guarantee of efficient utilization.

In this landscape, the ability to build internal benchmarks — even simple ones like cost per resolved task or time saved per transaction — becomes the real differentiator. The AI-RADAR community has long explored initiatives to map models, quantization, and serving strategies so that measured performance can be tied to concrete outcomes. On this topic, the /llm-onpremise section offers analytical frameworks for those looking to evaluate trade-offs without falling for the "cloud at all costs" dogma or the "server under the desk" approach with no criteria.

A compass for mindful deployment

The leaked audio is not just industry gossip; it signals that the entire ecosystem — vendors, integrators, consultants — is racing without a map. Finance directors, increasingly involved in technology decisions, will soon demand solid metrics. Those who can supply them, combining hardware telemetry with business logic, will gain a competitive edge.

In the meantime, reason suggests distrusting promises and equipping oneself with end-to-end monitoring tools, from the compute layer all the way to user experience. Only then will AI stop being a cost black hole and become a legible line item in the budget.