Quantization

• Your 3090 Isn't Missing Anything for FP8/NVFP4 Weight-Only Inference

  September 30, 2026 · 17 min · llm, vllm, quantization, gpu, cuda

  FP8/FP4 tensor cores are used only when the activations are quantized to FP8/FP4 as well. With weight-only (A16) quantization, both an RTX 3090 and Blackwell compute on BF16 tensor cores. This post shows the evidence in the vLLM source, in the SASS built for the 5090, and in measurements on a 3090 (kernel time, locked clocks, Nsight Compute). It states what the 3090 actually lacks (the prefill compute ceiling of W8A8/W4A4) and answers the objections I expect.

• Running 125B MoE on Three 3090s - vLLM Edition

  September 15, 2026 · 22 min · llm, vllm, quantization, qwen, gpu

  Running a 125B-A6B MoE carrying a 51.2B n-gram table on three RTX 3090s with a 262,144 context and 80 tok/s. Most of the KV cache can be placed in host RAM. Stock vLLM does not even boot in the first place, so I write about all the broken parts.

• vLLM Can Run INT5–7

  September 11, 2026 · 5 min · llm, vllm, quantization, qwen, gpu

  vLLM can natively execute INT5/6/7 weights. Here is an overview of how well they perform, along with results showing that quantizing embed_tokens and lm_head causes minimal loss.