minamism

Notes on local LLM inference, GPUs, and the machines underneath.

vLLM Can Run INT5–7
September 11, 2026 · 5 min · llm, vllm, quantization, qwen, gpu
vLLM can natively execute INT5/6/7 weights. Here is an overview of how well they perform, along with results showing that quantizing embed_tokens and lm_head causes minimal loss.
3.14M Context on 3x 3090: The Power of an Architecture That Puts KV Cache in Host RAM
September 10, 2026 · 9 min · llm, exllamav3, qwen, gpu
Qwen Sparse Attention reads the same number of tokens per step no matter how long the context is. Thanks to this, the KV cache can be placed in host RAM. With the same three GPUs, the upper limit went from 234k to 3.14M tokens.