TurboQuant-v3: Weight Compression for Accelerated LLM Inference

Google has released TurboQuant-v3, a new compression technique designed to reduce the memory footprint of large language model (LLM) weights. This approach focuses on compressing the model weights, unlike previous TurboQuant iterations that primarily targeted the KV cache.

TurboQuant-v3 uses a combination of group-wise INT4 quantization, AWQ scaling, FP16 outlier handling, and optional SVD correction. The goal is to significantly reduce VRAM usage, enabling the execution of larger models on hardware with limited resources, such as consumer GPUs.

The stated benefits include an approximate 4x memory reduction and a 2-3x increase in inference speed thanks to custom kernels. The technique is designed to be easily implemented, without requiring further model training.

For those evaluating on-premise deployments, there are trade-offs between performance, TCO, and compliance requirements. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.