vLLM Can Run INT5–7

When it came to weight quantization in vLLM, for a long time the only options were 4-bit, 8-bit, or leaving weights unquantized altogether.

4-bit is too coarse. KLD shoots up, and inference accuracy degrades more than most people realize. It works, but that is about as good as it gets.

On the other hand, 8-bit is huge. If you try to run Qwen3.8-27B in vLLM on two RTX 3090s, the 8-bit weights alone consume about 29.3 GiB. Add full-context KV cache, MTP, and the Vision Tower on top of that, and an OOM is unavoidable.

llama.cpp doesn’t (really) suffer from this issue. GGUF supports a wide range of bitwidths beyond just 4-bit and 8-bit.

In this post, I will cover how vLLM can now run INT5–7 thanks to the relatively recently merged humming-kernels, evaluate its performance, and discuss whether quantizing embed_tokens and lm_head should actually be avoided.

1. How INT5–7 Works in vLLM

With the merge of vLLM PR #46389, 5/6/7-bit support is now operational in vLLM.

That PR introduced HummingLinearKernel from humming-kernels, adding [2, 3, 4, 5, 6, 7, 8] to CompressedTensorsWNA16’s supported bitwidth mapping (WNA16_SUPPORTED_TYPES_MAP).

In other words, every bitwidth from 2 to 8 is now supported. There is no reason not to take advantage of this.

2. The Power of Intermediate Bitwidths

To see how strong INT5 and INT6 actually are, I downloaded 26 quantized variants of Qwen3.8-27B available on Hugging Face (including GPTQ, AWQ, AMD Quark, compressed-tensors, FP8, etc.) and compared them using the same evaluation setup.

For the benchmark, long texts of 8,192 and 32,768 tokens extracted from github-code-clean, proof-pile-2, and peS2o were used. Across a total of 589,788 next-token prediction positions, I measured Kullback–Leibler Divergence (KLD) and Top-1 agreement against the original BF16 model.

To ensure fairness, all models were dequantized back to BF16 and executed with the exact same F.linear implementation, eliminating discrepancies caused by kernel-level differences.

• Top-left: Clustered around 17–20 GiB. KLD ranges from 0.010 to 0.030 nats.
• Bottom-right: Clustered around 27–34 GiB. KLD ranges from 0.0006 to 0.0012 nats.
• Center:
  • Minachist/Qwen3.8-27B-INT5-Flat-5.7bpw-AutoRound (18.5 GiB / 5.73 bpw): KLD 0.00373, Top-1 98.10%
  • Minachist/Qwen3.8-27B-INT6-Flat-6.6bpw-AutoRound (21.3 GiB / 6.60 bpw): KLD 0.00151, Top-1 98.73%
  • Minachist/Qwen3.8-27B-INT6-Mixed-AutoRound (23.6 GiB / 7.30 bpw): KLD 0.00099, Top-1 98.94%

Looking at Top-1 agreement:

While 4-bit tops out around 97.2%, INT5-Flat achieves 98.10% at just 5.73 bpw, and INT6-Mixed reaches 98.94%, putting it within striking distance of 8-bit quality (99.1–99.3%).

Adding just 1 to 2 bits over 4-bit slashes quantization error by a factor of 3x to 10x. When 4-bit is unacceptable but 8-bit won’t fit (which is almost always the case), these intermediate bitwidths are practically the sweet spot.

3. What About embed_tokens and lm_head?

Qwen3.8-27B has a vocabulary size of 248,320 and a hidden dimension of 5,120. Leaving both in BF16 consumes 2.37 GiB for embed_tokens and another 2.37 GiB for lm_head, taking up a combined 4.74 GiB of VRAM. That is quite substantial.

The likely reason model creators often ignore this is an oversight in vLLM’s implementation. vLLM already has a kernel called CompressedTensorsEmbeddingWNA16Int that can execute quantized embeddings. However, in vllm/model_executor/models/qwen3_5.py (which Qwen3.8 also routes through):

self.embed_tokens = VocabParallelEmbedding(self.vocab_size, config.hidden_size)

Neither quant_config nor prefix is passed. As a result, attempting to load a quantized embedding crashes.

Should embed_tokens and lm_head Be Quantized?

I also benchmarked which approach yields better quality: keeping the vocabulary layers in BF16 while squashing intermediate layers down to INT4, or dropping the vocabulary layers to INT8 and using the freed-up 2.33 GiB to bump intermediate layers up to INT6 or INT7.

In this graph, the horizontal axis plots the size of the Transformer body alone, excluding embed_tokens, lm_head, MTP, and the Vision Tower. From these results, at least for this model, there is little justification for leaving embed_tokens and lm_head in BF16—dropping them to INT8 is clearly the right move. In fact, many GGUF creators routinely quantize these layers. Compared to BF16, this saves nearly 4.74 GiB.

4. Does Quantizing linear_attn (Recurrent Layers) to INT5/6 Diverge on Long Contexts?

People often claim that quantizing recurrent layers to INT5 or INT6 causes rounding errors to accumulate as context grows, eventually diverging over long sequences. I put that to the test as well.

In long contexts, token positions were divided into bins (0–512, 512–2k, 2k–8k, 8k–32k), and the average KLD for each bin was measured.

As the context lengthens, KLD actually decreases monotonically. Compared to the initial bin (0–512), KLD in the deepest bin (8k–32k) drops to 0.52x–0.62x.

Even in subsequent experiments extending measurements up to 65,536 and 131,072 tokens, the KLD ratio of INT6 relative to INT8 remained between 1.20 and 1.27 across all bins, showing zero sign of divergence.

My guess is that the rule of thumb that linear_attn.{in_proj_a,in_proj_b} should not be quantized was mistakenly generalized to all of linear_attn.

Summary

1. vLLM can natively execute INT5–7: Thanks to HummingLinearKernel, it works out of the box on sm75+ architectures without any extra builds.
2. Intermediate bitwidths are powerful: Having INT5/6/7 available between 4-bit and 8-bit is a huge deal. Increasing by just 1 bit brings a substantial jump in accuracy.

There is no longer any need to be constrained by the old conventional wisdom that quantization is just “4-bit or 8-bit”. If you want to maximize both context length and output quality within limited VRAM, stepping into the world of INT5–7 is definitely worth it.