vLLM Optimization Techniques: Diagnosing GPU, CPU, Memory, and I/O Bottlenecks

Optimizing vLLM requires more than finding a slow operation. A change to that operation must also make the request finish sooner. CPU execution, GPU execution, communication, and storage can overlap. An operation’s duration and its contribution to response latency are different quantities.

The measured examples used in this article are summarized first.

These values must not be combined into the result of a single configuration. Each section specifies the comparison conditions.

This model combines MoE, linear attention, and sparse attention. Custom patches also place KV in host RAM and mmap the PLE n-gram table. PLE is an embedding table whose row indices are calculated from token sequences. Stock vLLM does not read NVMe during inference for every model. The article distinguishes behavior specific to this configuration from investigation methods that apply to other models.

No new GPU experiments were run for this article. Input lengths, output lengths, and durations in commands are examples, not new measurements. CLI options and metric names depend on the version. Use the running image’s --help and /metrics as the reference.

1. Turn “slow” into a measurable symptom

Separate startup, TTFT, decode, and concurrent serving

A slow first token and slow subsequent tokens require investigating different intervals. Reducing startup compilation does not necessarily improve steady-state decode on an already running server.

Let N be the number of generated tokens, t0 the start of submission, and t1 and tN the arrival times of the first and last tokens. When individual token arrivals are observable, the following definitions apply.

TTFT = t1 - t0
Mean TPOT = (tN - t1) / (N - 1)   [N > 1]
Mean decode rate = (N - 1) / (tN - t1)

TPOT means time per output token. This differs from N / (response completion - t0), which includes TTFT and completion notification time. Keep the denominator consistent when comparing results.

Do not equate the number of SSE events with the number of tokens. Role-only events, usage, and termination notifications must be excluded. Speculative decoding, grouped streaming output, or proxy buffering can put multiple tokens in one event. Separate reasoning and answer content also matter. If per-token timestamps are unavailable, report “received chunk intervals” and cross-check against server token metrics.

Separate concurrency-1 speed from aggregate throughput

Higher concurrency can increase aggregate tok/s while increasing each request’s ITL. Improving one user’s response time and processing more requests in the same interval are different objectives.

Before investigating, specify the input length, output length, load, and metric to improve. If the target is concurrency-1 decode with a long context, maximum throughput on short prompts alone cannot answer the question.

2. Reason about dependencies rather than summing operation times

GPU launches are asynchronous. Previously launched GPU work can execute while the CPU prepares the next input. Communication between GPUs and file prefetch can overlap as well.

The critical path is the longest path through the dependencies leading to response completion. Shortening work outside that path does not shorten the response.

CPU:  [prepare input] [launch GPU work] [next preparation........]
GPU:                                 [compute..................]

With synchronization in between:
CPU:  [launch] [wait for GPU completion........] [next preparation]
GPU:           [compute......................]                   [next compute]
                                                 GPU waits until preparation finishes

A long cudaMemcpy or aten::to does not necessarily mean a large transfer. It may be waiting for earlier work on the same stream. Conversely, even a small transfer can delay subsequent launches if it stops the CPU from running ahead.

Investigate in this order:

1. Identify the interval in which the slow request spends time.
2. Determine whether the GPU is computing and whether the CPU is executing or waiting.
3. If something is waiting, identify what must finish first.
4. Construct a comparison that changes that dependency alone.
5. Remove the profiler and confirm that response time improves.

Utilization narrows the candidates. Dependencies and controlled comparisons establish the cause.

3. Fix the reproduction conditions before comparing

Conditions to save

Save the effective configuration in startup logs as well as the arguments. A requested fast path may fall back to another implementation. Adding an environment variable does not prove the optimization is active.

Distinguish three kinds of cache

Disabling prefix caching does not disable the OS page cache. Restarting a server does not clear the host page cache. Resetting prefix cache and creating cold file reads are different operations.

Use fixed prompts and seeds to compare steady-state decode. The September 23 comparison used 1 warmup followed by 3 measured runs. Cache misses on new text and drive stalls were evaluated separately.

That repetition count does not guarantee significance for a small difference or a reliable p99. Include raw observations, sample counts, and variation between runs. With few samples, p99 is effectively close to the maximum; check whether longer measurements show the same behavior. Do not discard slow runs without a reason.

4. Establish client and server baselines first

Reproduce with a small load

The following synthetic load targets an already running test server. Replace MODEL with the served model name and TOKENIZER with the local path of its tokenizer.

MODEL='your-served-model-name'
TOKENIZER='/path/to/tokenizer'

vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model "$MODEL" \
  --tokenizer "$TOKENIZER" \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --num-prompts 16 \
  --max-concurrency 1 \
  --request-rate inf \
  --ignore-eos \
  --seed 7 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,99 \
  --save-result \
  --save-detailed \
  --result-dir bench-results

This establishes a /v1/completions baseline. If production uses chat, tool calls, images, or structured output, also save workloads using the same path and template. Do not treat random-input results as production results.

--ignore-eos helps keep output length fixed. Check the actual output token count recorded in the results too. If requested and actual lengths differ, the denominator changes.

Repeating a benchmark with the same seed can produce prefix-cache hits. For uncached prefill, disable prefix caching on the test server or use the reset operation supported by that version. Verify the absence of hits in metrics as well. The vLLM benchmarking CLI documentation also describes the repeated-benchmark issue.

Compare different workload conditions

With --request-rate inf and a concurrency limit, completed requests allow new ones to be submitted. This is useful for capacity comparisons, but does not reproduce a production queue fed at a fixed arrival rate. Also test a specified arrival rate and record actual submission rate and outstanding requests. The client’s concurrency limit or CPU can prevent it from reaching the requested arrival rate.

Save server-side intervals too

curl -fsS http://127.0.0.1:8000/metrics > metrics-before.txt
# Run the benchmark between these snapshots
curl -fsS http://127.0.0.1:8000/metrics > metrics-after.txt

rg 'vllm:(num_requests_(running|waiting)|.*preempt|.*cache.*usage|request_(queue|prefill|decode)_time_seconds|time_to_first_token_seconds)' \
  metrics-after.txt

Track waiting requests, running requests, and KV utilization over time. Use before-and-after differences for queue, prefill, and decode histograms. Do not mix historical requests in cumulative histograms into the current measurement. Available metrics and their meanings are documented in vLLM Production Metrics.

If TTFT and queue time are both high, investigate queueing first. If concurrency 1 is normal but waiting requests keep accumulating under load, offered load may exceed capacity. A faster prefill kernel alone cannot establish an improvement without checking the load conditions.

Do not convert the server’s Avg prompt throughput log into per-request TTFT. In the history, a reporting window showed 4,206 tok/s, while an initial request of about 42k tokens had a TTFT of about 100 s. The prompt-token accounting time affected that build’s log. Measure client submission and first-token timestamps directly.

5. Observe GPU, CPU, RAM, and I/O during the same interval

Looking at the GPU for one request and the CPU for another does not compare the same stall. Match request start and end timestamps to each log. Begin with lightweight observation, then use a profiler.

Run the following examples in separate terminals. Set WORKER_PID to the worker PID visible from the host. The API server, engine, and GPU workers may be separate processes; inspect each. Stop the commands with Ctrl-C.

nvidia-smi \
  --query-gpu=timestamp,index,utilization.gpu,utilization.memory,memory.used,power.draw,clocks.sm,clocks.mem,temperature.gpu \
  --format=csv -lms 200

WORKER_PID=12345
pidstat -t -u -r -d -w -p "$WORKER_PID" 1

vmstat 1

iostat -x -z -y 1

pidstat and iostat are provided by sysstat. Some GPU fields return N/A depending on the product and driver. Polling every 200 ms does not necessarily reveal behavior finer than the driver’s internal sampling interval. Use traces for short stalls.

An observation needs another check before it becomes a conclusion

utilization.gpu measures the fraction of the sampling period during which a kernel executes. It does not measure the fraction of computational capacity used. Likewise, utilization.memory measures time during which device memory is read or written, not achieved bandwidth relative to peak. See the NVIDIA nvidia-smi definitions. A low mem% alone cannot quantify how much bandwidth remains.

The same distinction applies to CPUs. Low host-wide utilization can coexist with a sequential thread exhausting the CPU available to it. Conversely, high CPU utilization can be busy-polling for CUDA completion.

Separate execution from waiting using CPU stacks

Use py-spy, with support for the target Python version, to inspect where Python execution has stopped.

py-spy dump --pid "$WORKER_PID"

Check whether repeated observations find the same location. Distinguish tokenizer or metadata construction from waiting inside a copy or communication call. A Python stack alone may not reveal what happens inside a native extension.

Add a short CPU sample to investigate native code that is executing.

perf record -F 99 -g -p "$WORKER_PID" -o worker.perf.data -- sleep 15
perf report --stdio -i worker.perf.data

This uses perf record to sample execution; it does not aggregate time spent sleeping. Heavy CUDA polling does not justify a fix aimed at reducing CPU computation. Match waiting threads to wchan, stacks, and GPU traces. Attach permissions, available symbols, and sampling overhead depend on the environment.

Check container limits and NUMA placement

The host can have spare capacity while a container has CPU quotas or a memory limit. On a cgroup v2 host, find the process’s cgroup and collect values from the corresponding directory before and after the test.

cat "/proc/$WORKER_PID/cgroup"

# Replace with the actual cgroup path
CGROUP_DIR='/sys/fs/cgroup/path/to/the/container'
cat "$CGROUP_DIR/cpu.max" "$CGROUP_DIR/cpu.stat"
cat "$CGROUP_DIR/memory.current" "$CGROUP_DIR/memory.max"
cat "$CGROUP_DIR/memory.events" "$CGROUP_DIR/memory.stat"
cat "$CGROUP_DIR/memory.pressure" "$CGROUP_DIR/io.pressure"

Increases in nr_throttled and throttled_usec help investigate throttling by that cgroup’s CPU quota. Check ancestor limits too. Changes in memory.events fields high, max, oom, and oom_kill help investigate memory constraints. Definitions are in the Linux cgroup v2 documentation.

On multisocket systems, also inspect nvidia-smi topo -m, numactl --hardware, and numastat -p "$WORKER_PID". If the CPU near the GPU and the NUMA node holding host buffers differ, transfers take an additional path. When changing binding, record both CPU and memory placement and compare the same workload.

6. Use a profiler to identify why execution waits

Capture a few requests

On a supporting build, add the following when starting the test server. Include the normal model and parallelism options in the same command.

vllm serve /path/to/model \
  --profiler-config '{"profiler":"torch","torch_profiler_dir":"/tmp/vllm-traces"}'

curl -fsS -X POST http://127.0.0.1:8000/start_profile
# Send a small number of saved reproduction requests
curl -fsS -X POST http://127.0.0.1:8000/stop_profile

Open the traces in a viewer such as Perfetto. With multiple workers, inspect files from every rank. The startup configuration, start and stop APIs, and short captures are described in the vLLM profiling documentation. Older builds use different configuration, so do not apply the example without checking.

Use PyTorch Profiler to inspect operations and callers. Use Nsight Systems to examine CPU threads, CUDA APIs, streams, and communication in more detail. Once the question concerns memory access or computational efficiency inside a particular kernel, use Nsight Compute. End-to-end waiting and efficiency inside a kernel require different measurements.

Align events on the same timeline

1. What was launched immediately before the GPU gap?
2. Is the CPU preparing inputs in Python, inside a CUDA API, or communicating?
3. Is a copy H2D or D2H? What are its size and stream?
4. Does NCCL time represent data transfer or waiting for the sender?
5. How do the preceding stage’s completion, receive completion, and next launch correspond?

Summing CUDA times in an aggregate table does not give request latency. Streams and GPUs can overlap. Adding a CPU operation’s inclusive time to its children also counts work twice.

For custom host traces, use a monotonic clock such as monotonic_ns within one host. Record request ID, step, rank, start, and end. Monotonic timestamps from different hosts are not directly comparable. Matching communication by start time alone can associate a receive that began waiting early with the wrong step.

Do not report profiled tok/s as the final result

In the September 15 investigation, decode normally running at about 45 tok/s fell to about 24 tok/s under the profiler. On September 23, kernel time summed inside a full CUDA graph appeared as 12.15 ms, while normal ITL was 10.66 ms.

Traces were therefore used to identify ordering and candidates. GPU intervals were measured again with CUDA events around graph replay. Check the ending event after the measured work; do not introduce a new synchronize() at every step. Include the necessary dependencies if the interval covers work on other streams.

Kernel durations are not guaranteed to remain unaffected by profiling either. Collecting hardware counters can involve replay and cache control. Confirm the final effect with the profiler and detailed instrumentation removed.

7. Separate CPU execution from synchronization: omit gather in a test

Symptom: decode stops at about 45 tok/s

The September 15 configuration gathered the PLE table on the host. The path included returning GPU row indices to the CPU.

ids = flat_ids.to(device="cpu", dtype=torch.int64).numpy()

The transfer was small. However, synchronous D2H could make the CPU wait and delay launches in subsequent eager regions. Slow np.take gather or NVMe reads were other possibilities.

Establish the cause with a comparison that retains synchronization

A test-only implementation omitted selected work. Conditions were three RTX 3090 GPUs, PP=3, maximum context 262,144, --max-num-seqs 4, KV 550 MB, prefill chunk 512, QSA offload enabled, and prefix caching disabled. The comparison used concurrency 1, 256 output tokens, 1 warmup, and 3 measured runs.

A and D were nearly equal. Omitting gather and H2D did not produce a measurable improvement under these conditions. The difference between D and B identified synchronization limiting CPU run-ahead as a major cause.

B, C, and D do not generate correct output. They omit work to investigate the cause and potential improvement; they are not user-facing performance results. Changed output can also alter MoE routing and referenced pages, so this experiment alone cannot establish the final improvement.

Preserve correct processing and make a full graph possible

The actual fix retained gather. Before forward, it filled an existing device buffer with a fixed address. Forward read that buffer. Moving CPU work out of forward made full CUDA graph execution possible for decode.

The gather location and graph mode were compared separately in the same build.

Moving gather alone did not improve speed. It was a prerequisite for using a full graph. “Improvement from moving gather” and “improvement from the graph” must not be treated as independent additive gains.

The first article described the experiment as showing that optimizing gather had zero value. That wording was too broad. More precisely, omitting gather and H2D did not produce a measurable improvement in this steady-state decode comparison. It does not apply to prefill with frequent page-cache misses, discussed in section 11.

8. When both CPU and GPU constrain execution: PP receive ordering

Symptom: fusion has different effects under different inputs

Fusing hyper-connection GPU work shortened a short-prompt step by only about 0.4 ms. Under the 8k condition, it shortened the step by about 2 ms. Similar reductions in GPU work contributed differently to response time.

Cause: CPU preparation began after waiting for a receive

Non-first PP ranks called irecv_tensor_dict() at the start of Worker.execute_model. Its recv_object waited for metadata from the preceding stage. Input preparation and attention metadata construction began only afterward.

This CPU work took 3.2-3.5 ms per rank. Overlapping it with the preceding stage’s GPU work could reduce waiting.

Before:
Previous GPU [forward........]
Next CPU     [wait to receive][input and metadata preparation]
Next GPU                                                    [forward]

After:
Previous GPU [forward........]
Next CPU     [input and metadata preparation][receive]
Next GPU                                             [forward]

This is a schematic of ordering. Horizontal widths do not represent measured durations.

Fix and result: compare each change and their combination

Receive was deferred until immediately before forward. Send/receive matching and the device order of receive, forward, and send were preserved.

Environment switches in the same build were compared with prefix caching and QSA offload enabled. The baseline procedure used a short prompt, fixed seed, 400 output tokens, 1 warmup, and 3 measured runs. The table reports median ITL from the client harness used at the time.

Each change alone had a small effect, but together they substantially reduced latency. Overlapping CPU preparation still leaves a constraint if GPU work is long. Shortening GPU work still leaves a constraint if CPU preparation remains serialized.

This situation needs a comparison covering both changes independently and together. Adding separately measured effects does not predict the combined result.

The historical client measurement aggregated SSE event intervals after excluding intervals near the beginning. It was used for comparisons on the same path with MTP disabled, but did not independently verify individual token arrivals. These historical values must not be compared unconditionally with token ITL from MTP or another streaming implementation.

9. Investigate inside the GPU: bandwidth, computation, and small kernels

Do not infer decode traffic from total model size

MoE does not read every expert for every token. To estimate decode traffic, separate dense weights, selected experts, scales, KV, and other temporary data. Check quantization packing and the kernel that actually runs.

On September 15, safetensors headers and the number of selected experts gave an estimated weight traffic of about 5.83 GB/token. Using the RTX 3090’s specification bandwidth of 936 GB/s gives the following simple read time.

5.83 GB / 936 GB/s ≈ 6.23 ms/token
1 / 6.23 ms ≈ 160 tok/s

This calculation uses the historical figures. It excludes computation, dequantization, communication, KV, and CPU preparation. It also simplifies cache reuse and repeated reads. It is an optimistic weight-read reference for one request passing sequentially through PP stages, not a guarantee of attainable performance.

A difference from measured time must not all be classified as kernel inefficiency. Dividing by an interval that includes CPU waiting and GPU gaps reduces the apparent bandwidth.

Distinguish computation limits from memory limits

Even during continuous GPU execution, several internal constraints are possible. Inspect the actual kernel shape, dtype, operation count, and reads and writes at each memory level.

A high counter alone does not establish that an optimization will help. Evaluating an L2-served kernel against specification DRAM bandwidth uses the wrong reference. Evaluating INT4 dequantization using only BF16 matrix performance does too. Microbenchmarks also need the cache conditions, shapes, and layouts used in serving.

Inspect shape and duration per kernel

Once GPU execution remained on the critical path, large and small operations were measured separately.

These estimates divide weight bytes by duration. They are not DRAM bandwidth measurements from hardware counters. L2 caching, scales, and repeated reads require additional counters.

Large projections achieved high effective bandwidth, while small GEMVs were slow. For small operations, launch count, intermediate tensors, and reduction work can matter more than reducing data size.

Fusing one hyper-connection operation from 6 kernels into 2 reduced its duration from 32 µs to 13.3 µs. It executes 96 times per token. Simple multiplication gives (32 - 13.3) µs × 96 ≈ 1.80 ms of potential savings. How much appears in response latency depends on the CPU overlap described in section 8.

Avoid slowing prefill while optimizing decode

Small-batch decode and prefill GEMM have different shapes. That does not justify using the same tile.

The historical implementation used a 32 × 64 tile for the prefill down projection. A 64 × 64 tile took 176 µs in that kernel and slowed overall prefill by 4%. Do not make the adoption decision from decode alone. Also check long-prompt TTFT and peak activation memory.

10. Investigate memory capacity separately from memory speed

Break down VRAM use

Beyond weights, VRAM holds KV, activations, workspaces, CUDA graphs, and communication buffers. The PyTorch allocator can also retain unused memory.

import torch

print("allocated", torch.cuda.memory_allocated())
print("reserved", torch.cuda.memory_reserved())
print("peak_allocated", torch.cuda.max_memory_allocated())
print("peak_reserved", torch.cuda.max_memory_reserved())

This instrumentation belongs inside the worker. Running it in a separate process does not retrieve the target worker’s statistics. nvidia-smi and PyTorch statistics also cover different allocations.

Separate measurements after startup, loading, repacking, capture, at the prefill peak, and during decode. Identify the phase that causes OOM. Steady-state free memory alone cannot explain an OOM caused by prefill temporaries.

An early September 23 implementation increased reserved while leaving allocated and max_allocated unchanged. Idle GPU1 memory use rose by about 190 MiB. The cause was a changed allocation pattern during load_model. Removing repacking changed the reuse conditions for other allocations.

Adjusting allocation and release patterns improved this implementation. In general, not all of reserved - allocated is fragmentation; it includes reusable cache. Inspect allocation history with a memory snapshot when needed. Repeated empty_cache() calls do not necessarily resolve live tensors or split allocation regions.

Trace host RAM jumps to allocation sizes

Increasing QSA host KV caused a discontinuous jump in RAM use. The PyTorch pinned allocator in that experiment rounded allocation sizes up to powers of 2.

One layer needed about 4.239 GiB, which was rounded to 8 GiB. Repeating the allocation across 12 layers made the excess substantial.

The comparison used pinned_max_round_threshold_mb:2048 in that build. This diagnoses the PyTorch implementation used in the experiment; it is not a recommended setting valid for every version. The final distributed implementation instead allocated the required size directly and pinned it.

Inspect host memory with the following commands.

free -h
rg 'MemAvailable|Cached|Swap|Mlocked|Unevictable' /proc/meminfo
cat "/proc/$WORKER_PID/smaps_rollup"
cat "/proc/$WORKER_PID/io"

Simply adding RSS across workers counts shared pages repeatedly. Inspect PSS and the anonymous, file-backed, and shared components. Mlocked alone does not necessarily account for all CUDA-pinned memory. Cross-check allocator statistics, allocation logs, and host-wide memory use.

Pinned RAM cannot be reclaimed like ordinary file cache. Increasing KV capacity reduces the RAM available for PLE’s page cache. In this configuration, a capacity change affected storage I/O and TTFT.

Investigate host memory bandwidth even when capacity is sufficient

Resident data can still face contention from multiple workers reading or copying into staging buffers. This is separate from page faults and storage waits.

Measure gather and copy times against the same resident data. Compare NUMA placement and the number of workers reading concurrently. Collect CPU memory-controller counters where available. LLC miss counts alone do not establish DRAM bandwidth saturation.

GPU UVA reads from host memory involve host RAM, NUMA connections, PCIe, and dependent GPU loads. A fast host-side copy alone does not establish fast GPU reads.

Confirm KV pressure through preemption and recomputation

Insufficient KV can increase waiting and preemption. Alongside instantaneous utilization, inspect measurement-period preemption deltas, newly computed prompt tokens, and completion time.

Reducing --max-num-seqs or --max-num-batched-tokens can reduce concurrent work or temporary memory. It can also change throughput and TTFT. The vLLM tuning documentation describes the relationship between chunked prefill size, ITL, and TTFT. Measure both again under the target load.

11. For I/O, inspect page counts and waits as well as MB/s

Symptom: only the initial TTFT is high

The September 22 configuration used a PLE mmap table of about 95.4 GiB. Host RAM was about 125 GiB, with about 47.5 GiB pinned for QSA host KV. The whole table could not remain in page cache. fincore showed about 2.5 GiB resident.

Prefill read the required rows sequentially with np.take. When a referenced page was absent from page cache, the thread waited for it to be read. The GPU could be idle at this point even though GPU compute performance was not the cause.

Establish the cause by measuring the same gather on the CPU

Host-side measurements using the same mmap and row indices gave the following results.

The arithmetic was the same, but page state changed the duration substantially. This justified investigating faults and I/O before optimizing CPU computation.

PLE references 16 rows per token. Even a small amount of useful data can require separate random page reads. The drive measured about 4.5k IOPS for 4 KiB random reads at QD1 and about 36k IOPS at QD64 and above. QD is the number of outstanding I/O operations; IOPS is operations per second.

If the program waits for one read before issuing the next, NVMe queue depth stays low. Sequential-read GB/s for a large file does not predict this operation’s duration.

Request pages as soon as the row indices are known

For this PLE, every prompt row could be calculated from the input tokens alone. The fix calculated row indices at request admission and requested pages from another thread before prefill needed them. The gather calculation itself was unchanged.

Measurements on the same machine using previously unread vLLM Python source files of about 37k tokens were as follows.

The A/B inputs were different unread files with similar lengths and content types. They were not identical prompts with identical resident page sets. Do not derive a strict matched-condition speedup ratio from this table alone. The host gather and QD measurements also supported serial page faults as the major cause.

For another environment, define cold and warm conditions first. Use an isolated setup that can restore the same input to the same cache state, or assign multiple comparable inputs to A and B and reverse the order. Even unread text can share n-gram pages already in cache. Save residency and actual read volume too.

Resetting prefix cache alone cannot create this cold condition. Clearing cache across a shared host also changes unrelated workloads.

Match tail stalls to threads and drives

for task_dir in /proc/"$WORKER_PID"/task/*; do
  printf '%s ' "${task_dir##*/}"
  cat "$task_dir/wchan"
done

The history recorded folio_wait_bit_common during 100-780 ms decode stalls. This helped identify waits for page I/O completion. wchan alone does not identify the file or device; combine it with stacks, /proc/<pid>/io, and device statistics.

Under matching O_DIRECT 4 KiB random-read conditions, the original drive recorded a maximum of 970 ms and the comparison drive a maximum of 22 ms. These are maxima during individual measurement periods, not guarantees for entire product lines. The investigation needed to distinguish software improvements from long storage responses.

The following is an additional test against an existing file, not the script that produced those historical values.

TABLE_FILE='/path/to/existing/table.bin'

fio --name=table-randread \
  --filename="$TABLE_FILE" \
  --readonly --rw=randread --bs=4k \
  --direct=1 --ioengine=libaio --iodepth=1 --numjobs=1 \
  --time_based --runtime=30 --randrepeat=1 \
  --group_reporting --output-format=json --output=fio-qd1.json

Compare the same file and conditions with --iodepth=64 and a different output filename. Save IOPS, bandwidth, completion latency distribution, and actual depth. Check that the requested depth is achieved. O_DIRECT avoids the ordinary page cache; it does not reproduce the mmap cache hit rate or filesystem behavior of the real workload. See the official fio documentation for option definitions.

12. Separate average bandwidth from waits in PCIe, communication, and offload

Small transfer volume does not eliminate latency

Dividing bytes by bandwidth estimates ideal transfer time. In practice, when those bytes become known matters.

In QSA, the indexer in each layer selects the rows to read. The required K/V reads cannot begin until selection finishes. Spare average PCIe bandwidth over a second does not remove waits along the dependencies within each layer.

On September 23, a 32k configuration with QSA offload disabled served as a control. Without offload, step time stayed at about 12.5 ms for contexts 0, 8k, and 24k. The extra approximately 2 ms with offload was checked against kernel measurements. The relevant QSA kernel grew from about 30 µs to 190 µs per layer.

This control investigated the relationship between offload and context length. It did not have the same context capacity. Distinguish it from a normal serving-performance comparison.

The first article was wrong to state that PCIe reads could overlap computation on this path. Selected-row reads were part of the layer’s dependency chain, and measurements showed about 2 ms of additional time. The earlier optimization article also recorded this correction.

Host reads may exist without a visible transfer API

A GPU kernel can read pinned host memory directly through UVA without a separate large cudaMemcpy. The delay appears inside kernel time. Check host placement, read volume, and effective PCIe bandwidth too.

The same caution applies to NCCL across GPUs. A long communication kernel alone does not prove insufficient link bandwidth. If the sender is late, the receiver waits. Align the same step across ranks and distinguish data readiness from receive completion.

Inspect connectivity with nvidia-smi topo -m and lspci -vv. Link speed and width can differ between idle and load. Check the connection actually negotiated under load, rather than the slot’s specification. Collect NCCL debug logs for investigation, then restore normal logging for final comparisons.

At concurrency 1, PP latency depends on passing through each stage. With multiple requests, imbalance between stages can also affect throughput. TP splits computation while adding communication. Evaluate changes in parallelism using both concurrency-1 ITL and aggregate throughput at higher concurrency.

13. Construct comparisons that connect measurement to a fix

Write the hypothesis and its prediction before changing code

Construct a comparison that can disprove the hypothesis. Enabling every fast option and observing an improvement does not identify the cause.

Use single-variable comparisons and interaction comparisons appropriately

Start by toggling one implementation within the same build. For settings requiring a restart, include variation between restarts. The same build does not mean one continuously running process.

For small differences, test B → A as well as A → B. Avoid having temperature, cache, or background-load changes coincide with the setting change. When both changes are needed, as in section 8, compare their combinations.

Validate correctness through the path that changed

For kernel optimizations, compare reference outputs using the same input and weights. Check packing with bit-level round trips, top-k IDs and tie-breaking, and fusion’s rounding positions and tensor error. Cover prefill, decode, multiple requests, and different lengths.

If the server implementation changed, validation must go through that server. Evaluating the original checkpoint in Transformers does not exercise a vLLM patch and cannot validate it.

Exact generated-text matching is insufficient too. On September 23, mean absolute prompt-logprob difference was 0.001995 for old versus new and 0.002010 for repeated old-versus-old runs. Measure run-to-run variation, then combine direct tensor comparisons, distributions, and task results. That comparison alone does not prove equivalence for every input.

For speculative decoding, do not decide from acceptance rate alone. Include draft and verification cost, reserved KV, TTFT, and concurrent throughput. Measure speed with newly produced output tokens, not draft-token counts.

14. Record the conditions and constraints before closing the investigation

Run the final comparison without a profiler. Under the workload defined at the beginning, report the following before and after the change.

After an improvement, repeat the observations. Reducing CPU synchronization can make GPU execution the constraint. Increasing KV can make I/O the constraint. Work that did not matter in the initial investigation can become the next target.

The following sequence makes the investigation easier to assess.

1. Identify whether startup, TTFT, decode, tails, or queueing under load is slow.
2. Reproduce at concurrency 1 and align client timings with server intervals.
3. Observe GPU, CPU, RAM, and I/O during the same request.
4. Use traces to identify waiting dependencies and write a hypothesis and prediction.
5. Run comparisons that distinguish the hypotheses.
6. Implement a fix preserving correct processing and measure again without profiling.
7. Recheck speed, capacity, and correctness at the target lengths and concurrency.

Summary

• Separate startup, TTFT, decode, throughput, and tails. Do not compare tok/s with different denominators.
• Do not infer causes from utilization alone. Distinguish CPU execution from synchronization, GPU computation from communication waits, and memory capacity from bandwidth.
• Record prefix-cache, page-cache, and compilation states separately.
• Use traces to form hypotheses. Narrow down causes with controls separating synchronization from work and with individual and combined changes.
• Investigate memory peaks and allocation phases. Increasing pinned RAM can also affect I/O and TTFT.
• Measure again without profiling. Check correctness, prefill, concurrent throughput, and retained capacity too.