A recent study evaluated the impact of quantization on the user capacity of the Qwen3-32B language model. The results indicate that using INT4 quantization allows serving 12 times more users compared to using the BF16 format, with a minimal reduction in accuracy (1.9%).

Benchmark Details

The benchmark was conducted using an H100 GPU, comparing the performance of the Qwen3-32B model with different precisions: BF16, FP8, INT8, and INT4. Over 12,000 MMLU-Pro questions were used, and 2,000 inferences were performed to evaluate accuracy and user capacity.

Results

The results show a significant increase in user capacity when moving from BF16 to INT4. Specifically, the capacity increased from 4 concurrent users (with BF16) to 47 users (with INT4) with a 4k context window. This increase is directly related to the memory savings achieved through quantization.

For those evaluating on-premise deployments, there are trade-offs to consider between accuracy and computational resources. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.