Optimizing LLMs on Local Hardware: The Qwen3.6-27B and RTX 3090 Case

Running Large Language Models (LLMs) on local infrastructure presents an ongoing challenge for companies aiming to maintain control over their data and optimize operational costs. In this context, recent work has demonstrated significant progress in optimizing the Qwen3.6-27B model on a single NVIDIA RTX 3090 GPU, a common hardware configuration for self-hosted deployments. The primary goal was to push the limits of the context window and enhance stability for workloads involving tool-agent interaction.

Current results indicate the ability to manage a context of approximately 218,000 tokens with a throughput of 50-66 tokens per second (TPS) for text and code, and about 198,000 tokens with vision capabilities at 51-68 TPS. These figures, while entailing a slight drop in throughput compared to previous configurations, represent a significant increase in context window and operational stability, particularly for tool calls generating large outputs, which now complete without Out Of Memory (OOM) errors.

Technical Details and the Solution to Instability

The improvement in stability was the result of an in-depth analysis of a persistent problem: previously, extended tool outputs, up to approximately 25,000 tokens, consistently caused system crashes. The root cause was identified as a Genesis patch (PN12), designed to mitigate a memory issue, which was not being correctly applied in vLLM dev205+ versions. Despite the system reporting successful patch application, the underlying code path remained unchanged.

The core of the problem lay in an "anchor drift" within the patch, meaning a shift in anchors that prevented the correct modification of the code. Once this specific flaw was resolved, OOM errors during tool prefill disappeared, making configurations with much larger contexts usable. The solution has been documented and made available via a pull request on GitHub, providing a valuable reference for the community developing and managing LLMs in local environments.

Implications for On-Premise Deployments

This type of optimization is of fundamental importance for CTOs, DevOps leads, and infrastructure architects evaluating self-hosted alternatives versus cloud solutions for AI/LLM workloads. The goal is not simply to maximize throughput or context length in isolation, but to balance both aspects to ensure a smooth and reliable user experience on specific hardware like a single RTX 3090. The ability to handle contexts over 200,000 tokens with usable throughput and stable tool-agent workloads is a key factor for enterprise applications requiring complex and data-sensitive processing.

It is important to note some limitations. There is still a second "memory cliff" around 50-60K tokens for single-prompt workloads on a single GPU. However, this limitation does not apply when using tensor parallelism, for example with two RTX 3090s. Results also depend significantly on Quantization and the specific model configuration, highlighting the need for careful optimization. For those evaluating on-premise deployments, AI-RADAR offers analytical frameworks on /llm-onpremise to assess these trade-offs, providing a solid basis for informed decisions.

Future Outlook and Considerations for AI Infrastructure

Optimization work continues to be a cornerstone for the widespread adoption of LLMs in enterprise contexts with stringent data sovereignty and control requirements. The ability to run complex models like Qwen3.6-27B with extended contexts on accessible hardware such as a single RTX 3090 opens new opportunities for developing innovative AI applications, from code generation to legal document analysis, while keeping data within the corporate perimeter.

These advancements highlight the dynamic nature of the LLM ecosystem and the importance of Open Source collaboration in overcoming technical challenges. The community continues to explore how best to balance context and throughput on different hardware configurations, such as the RTX 3090 and 4090, constantly pushing the boundaries of what can be achieved with local infrastructures.