A Silent Bug Compromised Qwen3.5 35B A3B

The Qwen3.5 35B A3B model, an uncensored version developed by HauhauCS, garnered attention for its advanced architecture, which includes a Mixture of Experts (MoE) design with 256 experts, a hybrid DeltaNet + Attention structure, and 40 layers. Recognized for its ability to process recent information and its compatibility with consumer hardware like an RTX 3060 12GB GPU, it established itself as an interesting resource for local Large Language Model (LLM) deployments. However, despite its promising features, the model exhibited anomalous behavior: in prolonged conversations, it tended to lose context, repeat phrases, and generate fragmented code or code with incoherent comments, a phenomenon described as “philosophizing.”

This instability, particularly evident with long prompts, prompted a researcher to delve deep into the model's weights. The objective was to understand the cause of these deviations and restore the model's full functionality for applications requiring coherence and reliability, especially in on-premise usage contexts where performance control is crucial.

The Discovery and Fix of an Architectural Flaw

After two weeks of in-depth analysis, the researcher identified the root of the problem in two specific tensors, ssm_conv1d.weight, located in blocks 36 and 37 of the architecture. These tensors showed an anomalous scale, approximately 60% higher than normal (σ=0.102 compared to a median of 0.063). This discrepancy was attributed to the functioning of the AdamW algorithm, which, in the presence of rare experts in the final layers, can apply an excessively high effective learning rate, causing weight drift. In a recurrent architecture like DeltaNet, such drift compromises the model's hidden state, leading to rapid context loss after just a few tokens.

The implemented solution was straightforward: the two problematic tensors were scaled back to their normal values, without altering the other 489 tensors, whose scale is intrinsic to the architecture (such as gate_inp). Interestingly, a comparative analysis of the Gemma 4 26B A4B model revealed no similar issues, with all tensor scales correctly aligned. This discovery underscores the complexity of fine-tuning and optimizing LLMs, especially those combining MoE architectures with recurrent components.

Implications for On-Premise Deployments and Data Sovereignty

The results of the fix were significant: an 88.6% error reduction. The model now maintains coherence in long conversations, generates functional code, and has eliminated the “philosophizing” behavior. This enhanced stability is of paramount importance for CTOs, DevOps leads, and infrastructure architects considering LLM deployment in self-hosted or air-gapped environments. The ability of a model to run reliably on hardware like an RTX 3060 12GB, combined with its improved stability, offers a viable alternative to cloud-based solutions, with direct implications for Total Cost of Ownership (TCO) and data sovereignty.

For those evaluating on-premise deployments, LLM stability and efficiency are key parameters. A model that requires fewer resources to maintain coherence reduces operational costs and improves user experience. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate the trade-offs between self-hosted and cloud solutions, considering factors such as compliance, latency, and throughput. The ability to run complex models locally with predictable performance strengthens the strategy for corporate data control and security.

Lessons Learned and Future Prospects for Hybrid LLMs

The Qwen3.5 35B A3B case offers a valuable lesson: even a single bug in a few tensors can lock the potential of a complex, large-scale architecture. This highlights the need for rigorous verification and careful calibration, especially for LLMs that combine MoE architectures with recurrent components like DeltaNet or Mamba. Developers and implementers are advised to carefully examine the last blocks of these models, as AdamW might have silently introduced anomalies.

The Open Source community plays a crucial role in this refinement process, enabling the collaborative identification and resolution of issues that might otherwise remain hidden. The availability of models in GGUF format, along with optimized configurations for tools like LM Studio, facilitates adoption and optimization for local inference, pushing forward innovation in the field of on-premise LLMs. This episode reinforces the importance of a methodical approach to model validation, ensuring that their full potential is unlocked for enterprise applications.