Bonsai promises a ternary 27B LLM that retains 98.2% of the original model's intelligence. The whitepaper tells a different story. Ternary Bonsai 2 27B scores 52.8 on Terminal-Bench 2.1 and 60.8 on SWE-bench Verified. The document compares those figures with 69.7 and 80.6 attributed to Qwen3.5-27B, concluding that the model retains roughly three quarters of full-precision performance. But that name appears to be a typo, because the values would belong to Qwen3.8. Using the correct Qwen3.5 numbers changes the picture: 52.8 versus 41.6 (126.9%) on Terminal-Bench and 60.8 versus 72.4 (84.0%) on SWE-bench.
This is not just a naming issue. For anyone treating Quantization as a lever to reduce VRAM and run self-hosted models, the gap between a 98.2% retained-intelligence claim and three quarters on agentic coding is the core of the decision. Terminal-Bench and SWE-bench measure long-horizon, tool-driven software engineering capability, not simple language metrics. If hardware cost drops but the ability to complete software engineering tasks declines more than expected, the VRAM savings can be offset by a less favorable TCO and longer correction cycles.
The episode signals a structural shift: Quantization is no longer judged only by language benchmarks, but by agentic ones. Vendors that publish headlines without transparent baselines and typo corrections feed an opaque comparison. Those that document the original models precisely can turn technical honesty into a commercial advantage, especially on-premise where choosing a ternary quantization affects hardware design and deployment sustainability.
One concrete point remains: the whitepaper is on GitHub, page 7. Anyone evaluating the model can check the numbers without going through the headline.
๐ฌ Comments (0)
๐ Log in or register to comment on articles.
No comments yet. Be the first to comment!