The divergence between the results published by Unsloth and those of the ISTA team does not stem from parameter tuning but from a constraint on the number of tokens the model can generate. ISTA responded clearly: no hyperparameters changed, standard settings for a 763B-parameter model with a 1M context window. In ISTA's tests the context is 128K and the reasoning effort is 50. Unsloth, by contrast, uses a 32K token reasoning budget. ISTA allows up to 128K output tokens. This compute difference explains the score variance, not an alleged inferiority of the model.

A 32K token budget forces the model to compress reasoning into a few thousand tokens, reducing the number of intermediate steps and the ability to explore long chains. For a model designed for 1M contexts, this limit seems inconsistent: if the goal is to handle long documents or complex problems, the bottleneck should not be precisely in the amount of expressible reasoning. At the same time, a 128K output token budget is not a neutral detail: every generated token consumes additional VRAM for the context cache and lengthens inference time.

The comparison involves the GSQ-RCO quantization versions of DeepSeek-V4.1-Flash-1017. Quantization generally reduces memory footprint, but it does not eliminate the fact that the output window chosen in benchmarks affects computational cost and latency.

The case signals something structural for anyone evaluating self-hosted LLMs. Published numbers are not absolute snapshots: they are the result of test decisions that imply trade-offs among quality, cost, and hardware. A benchmark with a reduced token budget can penalize a model that, on a cluster with adequate VRAM and different production constraints, would deliver different results. This is especially true for models with hundreds of billions of parameters, where the difference between 32K and 128K tokens is not merely theoretical but directly impacts TCO and infrastructure sizing.

ISTA invites anyone with compute resources to reproduce the results using the same settings. The code is on GitHub, the papers are on arXiv. In the past, similar models were benchmarked by third parties like Benjamin Marie. The request is not a defense: it is a way to make benchmarks more readable. If the output budget is not standardized, every comparison will continue to measure not so much model quality as the compute resources one is willing to grant it.

Meanwhile, the team is rerunning the comparisons in its own environment: a move that, more than a rebuttal, points to the real issue: for models at this scale, inference policy is an integral part of the model itself.