The news comes from the SGLang-V100 repository and has the feel of an artisanal test, but it touches a raw nerve in on-premise deployment. The RadixArk/Qwen3.8-Flash-Next-NVFP4 model runs on four V100 32GB with a full 256k-token context. The system offloads just over 50GB of ngram mechanism to system RAM and, according to the published data, maintains prefill around 4,000 tokens/s and decoding around 60 tokens/s to the end of the context.

The details deserve attention because they break a common assumption: long contexts are not exclusive to the newest GPUs. Performance degrades unevenly. At 10k tokens, time-to-first-token without multi-token prediction is 2,154 ms and output is 59.93 tokens/s; enabling MTP raises output to 81.39 tokens/s but TTFT worsens to 2,307 ms. At 250k tokens, TTFT is 63,451 ms without MTP and 66,587 ms with MTP, while output drops to 56.85 and 62.58 tokens/s respectively. Decoding stays near the stated 60 tokens/s, but startup cost becomes the real bottleneck.

Concurrency also tells a more nuanced story. With a single stream, MTP lifts output from 55.12 to 74.82 tokens/s (+35.7%). With 8 streams, aggregate throughput rises to 172.98 tokens/s without MTP and 194.13 with MTP, but TTFT exceeds 8 seconds. With 16 streams, output with MTP falls to 183.87 tokens/s versus 239.36 without MTP, a negative 23.2% differential. MTP accept length remains around 3 tokens per step, a sign that speculative gains have limits under concurrent pressure.

For teams operating V100 fleets, these numbers have an immediate economic reading. It is not about beating an H100, but about understanding whether already amortized hardware can cover a class of workloads: long-document analysis, transcript review, extraction from extended corpora. With 128GB combined VRAM and offload to RAM, feasibility depends on the balance between system memory, acceptable latency, and number of concurrent requests. TCO shifts: the cost of a RAM upgrade and serving tuning can be far lower than new GPUs, but the price is paid in queueing and response variability.

Structurally, support in SGLang-V100 shows that the framework ecosystem is extending the hardware tail. Techniques like NVFP4 quantization and RAM offload do not close the generational gap, but they redefine the usage perimeter. For data centers handling sensitive data, this means keeping inference within their own boundaries longer, without forcing migration to cloud APIs. It is not a free path: the concurrency table shows that aggregate throughput grows, but per-request latency explodes, and the batch-versus-interactive choice becomes explicit.

For teams evaluating on-premise deployment, AI-RADAR provides analytical frameworks at /llm-onpremise to weigh these trade-offs, avoiding reducing the choice to a single metric. The number not to miss is that -23.2% at 16 streams: under real load, the same technique that accelerates a single stream can become a cost. That is where the difference between a forum benchmark and an infrastructure decision is played.