The quest for ever-larger language models collides with a physical constraint: GPU VRAM. Recent work on Ornith-1.0-35B, a 35-billion-parameter LLM, shows that aggressive quantization can turn an otherwise cumbersome model into a single-GPU solution without entirely sacrificing response reliability.

From BF16 to Q3_K_M: a drastic cut

The llama-quantize tool brought weights from a BF16 representation at 16.01 BPW down to 3.87 BPW in Q3_K_M format. The result is a 16.8 GB GGUF file on disk, which loads into roughly 17 GiB of VRAM. Compared to Q4_K_M (21.2 GB), the reduction is 21%; against Q8_0 (36.9 GB), memory usage is halved. For those working with 24 GB consumer GPUs, this compression opens the door to local inference without multi-GPU setups.

The community is well aware of the pitfalls of aggressive quantization: coherence loss, increased hallucination, degraded coding performance. To assess Q3_K_M, the author built a top-64 next-token KL divergence probe, comparing the quantized model’s probability distribution against the original BF16 on 32 coding prompts. The mean divergence was 0.366, rising progressively for less extreme quants: 0.086 for Q4, 0.035 for Q5, 0.017 for Q6. Meanwhile, top-1 agreement (exact token match) drops to 84.4% for Q3_K_M, versus 100% for Q6_K.

Serving performance and a key fix

Single-GPU inference with llama.cpp’s CUDA server yielded about 240 tokens per second single-stream, peaking at ~493 tok/s under 16 concurrent requests. The p95 time-to-first-token latency sits at 78 ms with a single connection – numbers that are ample for interactive local applications, especially since no enterprise accelerators are required.

A detail that surfaced during testing reveals a common trap: with llama.cpp’s reasoning mode left on/auto, short coding prompts can exhaust the response budget in internal reasoning content, returning empty final answers. Setting REASONING=off in the serving scripts restored correct behavior, confirmed by the full 14/14 behavior suite passing on a 16-slot serving profile. It’s the kind of bug that can turn into a diagnostic nightmare in production, and documenting it alongside the quantized models adds a layer of practicality to the repository.

The on-premise scenario: when every gigabyte counts

For organizations evaluating on-premise LLM deployments, the accuracy-vs-hardware trade-off is central. Q6_K guarantees maximum fidelity but demands 28.5 GB of VRAM, out of reach for many consumer cards and even some professional workstations. Q3_K_M, at 17 GB, can run on an RTX 4090 or an A4000, enabling data and inference to remain entirely on-premise. The precision loss – 16 points of top-1 agreement – must be weighted by application domain: in coding tasks it may be tolerable; in contexts where lexical correctness is critical, less so.

This is not a binary choice. Tools like the KL probe described provide a reproducible metric to assess whether a given quantization suits a specific use case, and the repository offers the entire reference ladder (Q4, Q5, Q6, Q8) for direct comparison. The approach, tested also via a LoRA smoke test to validate the training stack, points in a direction: on-premise operators can rely on objective metrics rather than intuition or generic benchmarks. The fact that the 14/14 behavioral suite passed with 16-slot serving further suggests that the degradation is predictable rather than chaotic.

Outlook

The author notes ongoing work on quantization for the 397-billion-parameter variant and on performance improvements for current quants. If the jump from 35B to 397B follows the same logic, the compression-reliability trade-off will become even tighter. Meanwhile, for those seeking a concrete starting point, the HuggingFace repository bundles quantized models, monolithic serving scripts, and OpenAI-compatible correctness checks: a reproducible mini-lab that reduces the chance of production surprises.