minamism
Notes on local LLM inference, GPUs, and the machines underneath.
- Running 125B MoE on Three 3090s - vLLM Optimization Edition
Qwen3.8-Flash-Next ran at 80 tok/s last time. Once it went into real use, first-request TTFT was 100 seconds, prefix caching cut the context by 30%, and decode in the production configuration was in the 60s, not 80. This covers the 11 patches added to reach 95-99 tok/s single-stream, 243 tok/s with four streams, image input and MTP, and the approaches that were tested and not adopted.
- Running 125B MoE on Three 3090s - vLLM Edition
Running a 125B-A6B MoE carrying a 51.2B n-gram table on three RTX 3090s with a 262,144 context and 80 tok/s. Most of the KV cache can be placed in host RAM. Stock vLLM does not even boot in the first place, so I write about all the broken parts.
- vLLM Can Run INT5–7
vLLM can natively execute INT5/6/7 weights. Here is an overview of how well they perform, along with results showing that quantizing embed_tokens and lm_head causes minimal loss.
- 3.14M Context on 3x 3090: The Power of an Architecture That Puts KV Cache in Host RAM
Qwen Sparse Attention reads the same number of tokens per step no matter how long the context is. Thanks to this, the KV cache can be placed in host RAM. With the same three GPUs, the upper limit went from 234k to 3.14M tokens.