Cuda

• Running 125B MoE on Two 32 GB Cards - vLLM Optimization Edition

  September 30, 2026 · 22 min · llm, vllm, qwen, gpu, cuda

  A user with two RTX 5090s asked whether Qwen3.8-Flash-Next runs on two 32 GB cards. The 62.3 GiB of weights do not fit, so the least used routed experts stay in host RAM, 3 GiB per card. With the choice of those experts, one expert table that spans VRAM and host RAM, and moving experts at run time, decode is 91.5 tok/s against 97.7 tok/s without the offload (measured on RTX 3090s).

• Your 3090 Isn't Missing Anything for FP8/NVFP4 Weight-Only Inference

  September 30, 2026 · 17 min · llm, vllm, quantization, gpu, cuda

  FP8/FP4 tensor cores are used only when the activations are quantized to FP8/FP4 as well. With weight-only (A16) quantization, both an RTX 3090 and Blackwell compute on BF16 tensor cores. This post shows the evidence in the vLLM source, in the SASS built for the 5090, and in measurements on a 3090 (kernel time, locked clocks, Nsight Compute). It states what the 3090 actually lacks (the prefill compute ceiling of W8A8/W4A4) and answers the objections I expect.