There’s a Reddit post that packs more punch than many white papers. Under the handle KriptacMessage, a user pushed an AMD Radeon AI Pro R9700 – a 32GB VRAM card – to run two Qwen3.6 models at 27B and 35B parameters, with context windows reaching 262,144 tokens. All on a single GPU, no load distribution, and at speeds that not long ago would have screamed for multi-GPU setups.

The secret sauce is aggressive INT4 quantization (W4A16, group_size 32) pulled from community-contributed Avesed weights on Hugging Face. The 35B at FP8 simply won’t fit 32GB with any usable context length. With INT4, though, billions of parameters and a generous context pool coexist. The benchmarks tell it best: the MoE 35B scales from 61.4 tok/s decode at 4k depth down to 49.5 tok/s at 150k, with prefill dropping accordingly, over a 262k token KV pool. The dense 27B leverages four speculative tokens and holds 59.6 tok/s decode up to 50k, then eases to 53.7 tok/s at 100k. All recorded on a self-hosted Podman container with vLLM.

This is no lab trick – it’s a concrete reference for anyone considering on-prem inference, from dev teams to smaller enterprises. The R9700, a professional workstation GPU, resembles the kind of hardware already sitting in edge servers or compact racks. Having a LLM with 100k+ token context running locally shifts the “local AI” benchmark way beyond tiny 7B models. Yes, INT4 quantization comes with a quality trade-off, but for many business use cases that’s acceptable, especially when weighed against full data control and zero recurring cloud API costs.

The devil, as always, is in the details. The user had to manually fix the tokenizer.json in the INT4 checkpoint, where truncation limits of 512 tokens from calibration broke the model’s vision capabilities. A quick tweak that lays bare the reality: the path is viable, but it needs someone comfortable with containers, config files, and startup scripts. There's no “run” button. Yet it's precisely this community of tinkerers that is laying the groundwork for a more robust on-prem offering. The custom vLLM image (stilldeadcode/vllm-radiance) pre-tuned for Radeon GPUs and packed with the right flags shows how open-source software is narrowing the gap with mature NVIDIA stacks.

For organizations watching on-prem deployments strategically, these figures signal a convergence. AMD, largely absent from production ML radars, is delivering hardware that can handle mid-sized model inference with enormous context lengths. Tensor parallelism set to 1 means no expensive NVLink or high-speed interconnects are needed, lowering the entry barrier. Total Cost of Ownership benefits from more accessible card pricing and modest power budgets.

The author is already eyeing Qwen3.8, a nod to the blazing iteration speed in this field. A reminder: on-prem isn’t a finish line but a continuous optimization race, where every token per second wrangled with ingenuity counts. The Radeon R9700 shows it’s a race worth running.