BLOG

Measured on Blackwell: Gemma-4 decode vs vLLM

A controlled comparison with vLLM on sm_120 — methodology and roofline analysis included

Batch-1 decode on an RTX PRO 6000 Blackwell: plow fp8 beats vLLM fp8 on Gemma-4-26B MoE from 16k to 128k context and cuts 31B dense latency by 41% vs vLLM bf16 — with the not-yet-tuned paths called out.

−41%
Gemma-4-31B dense decode vs vLLM bf16 — at every measured depth, 1k–32k
−21.2%
Gemma-4-26B MoE decode vs vLLM fp8 at 128k context (8.25 vs 10.48 ms/token)
+0.9%
serving overhead over the raw kernel on 12B — below measurement noise on 26B and 31B

Methodology

Every number below was collected under the same protocol:

  • Same box, same model, same quantization. One NVIDIA RTX PRO 6000 Blackwell (sm_120, 96 GB, 188 SMs, CUDA 13.0), head-to-head against vLLM 0.25.1 on identical checkpoints.
  • Correctness before the clock. A configuration earns a timing row only after passing a correctness gate — token-identity against the HuggingFace reference, or a bit-exact oracle against the f32 path. If an optimization changes the tokens, it doesn't ship.
  • Batch-1 decode, 112 timed tokens per point, measured at fixed context depths. Batch-1 is the single-user latency case — the one that matters for interactive and voice workloads, and the one we've optimized first.

Gemma-4-26B-A4B (MoE): decode results

At 1k context, vLLM fp8 measures 5.76 ms/token against plow's 5.94 — a 0.18 ms (3.2%) difference in vLLM's favor. At every deeper depth measured, plow's fp8-KV path is faster: 6.17 vs 6.82 ms at 16k (−9.5%), 6.45 vs 7.74 at 32k (−16.7%), 8.25 vs 10.48 at 128k (−21.2%). It is also below vLLM's fp8-KV variant at 16k, 32k, and 128k (6.62, 7.28, 8.46 ms respectively).

Gemma-4-26B-A4B decode latency vs context

ms per token, batch-1 · lower is better · RTX PRO 6000 Blackwell

plow fp8-KV vLLM fp8 vLLM fp8-KV
0 3 6 9 12 1k 16k 32k 128k context length (tokens) vLLM fp8 · 1k · 5.76 ms/token vLLM fp8 · 16k · 6.82 ms/token vLLM fp8 · 32k · 7.74 ms/token vLLM fp8 · 128k · 10.48 ms/token vLLM fp8-KV · 1k · 5.92 ms/token vLLM fp8-KV · 16k · 6.62 ms/token vLLM fp8-KV · 32k · 7.28 ms/token vLLM fp8-KV · 128k · 8.46 ms/token plow fp8-KV · 1k · 5.94 ms/token plow fp8-KV · 16k · 6.17 ms/token plow fp8-KV · 32k · 6.45 ms/token plow fp8-KV · 128k · 8.25 ms/token vLLM fp8 · 10.48 vLLM fp8-KV · 8.46 plow · 8.25 −21.2%

n=112 timed tokens per point, correctness-gated. Context axis is plotted at measured depths, not to linear scale.

The bf16 columns show the same crossover pattern without quantization in the picture: vLLM bf16 is ahead at 1k (7.61 vs 8.05 ms), and plow bf16 is below it at every deeper depth — 8.44 vs 8.64 at 16k (−2.3%), 8.94 vs 9.57 at 32k (−6.6%), 11.58 vs 12.34 at 128k (−6.1%).

Gemma-4-26B-A4B decode latency vs context — bf16

ms per token, batch-1 · lower is better · RTX PRO 6000 Blackwell

plow bf16 vLLM bf16
0 4 8 12 1k 16k 32k 128k context length (tokens) vLLM bf16 · 1k · 7.61 ms/token vLLM bf16 · 16k · 8.64 ms/token vLLM bf16 · 32k · 9.57 ms/token vLLM bf16 · 128k · 12.34 ms/token plow bf16 · 1k · 8.05 ms/token plow bf16 · 16k · 8.44 ms/token plow bf16 · 32k · 8.94 ms/token plow bf16 · 128k · 11.58 ms/token vLLM bf16 · 12.34 plow · 11.58 −6.1%

Same protocol as the fp8 chart: n=112 timed tokens per point, correctness-gated, ordinal context axis.

Gemma-4-26B-A4B decode, ms/token (batch-1) — full measured data; bold marks the fastest in each precision class
ctxplow bf16vLLM bf16plow fp8-KVvLLM fp8vLLM fp8-KV
1k8.0477.6105.9435.7605.920
16k8.4398.6406.1736.8206.620
32k8.9429.5706.4507.7407.280
128k11.58412.3408.25410.4808.460

Gemma-4-31B (dense): decode results

On the dense 31B, plow's fp8 (w8a8) decode measures 26.35, 26.47, 27.42, and 28.97 ms/token at 1k, 4k, 16k, and 32k — 41% below vLLM bf16 at each depth. Against vLLM's own fp8 path, the comparison shifts with context:

  • 1k–4k: within 3% of vLLM fp8 (+2.9% and +1.2%).
  • 16k–64k: plow ahead — −1.4% at 16k, −3.0% at 32k, and 31.42 vs 31.99 ms at 64k.
  • 128k: within 0.9 ms of vLLM fp8; below vLLM fp8-KV (38.63 ms) and vLLM bf16 (55.46 ms).

Gemma-4-31B w8a8 decode latency vs context

ms per token, batch-1 · plow, fp8 weights and activations · RTX PRO 6000 Blackwell

0 10 20 30 1k 4k 16k 32k 64k context length (tokens) plow w8a8 · 1k · 26.35 ms/token plow w8a8 · 4k · 26.47 ms/token plow w8a8 · 16k · 27.42 ms/token plow w8a8 · 32k · 28.97 ms/token plow w8a8 · 64k · 31.42 ms/token 26.35 26.47 27.42 28.97 31.42

A 64× increase in context (1k→64k) raises per-token latency by 19%. Reference points at 64k: vLLM fp8 measures 31.99 ms; at 128k, vLLM fp8-KV 38.63 and vLLM bf16 55.46 ms.

Prefill is the not-yet-tuned half of this model. The w8a8 path reduced time-to-first-token by 36–42% relative to our bf16 path, and at 32k it measures 16% below vLLM bf16; the remainder of that path awaits the auto-tuner described below.

Measured effect of each fp8 lever (Gemma-4-31B unless noted)
levermeasured effect
w8a8 (fp8 weights + activations)−48% GEMM time vs bf16 · −30 to −34% end-to-end prefill
fp8-KV (e4m3 KV cache)KV bytes −50% · 31B max batch 4 → 7–8

One accuracy note on fp8-KV, since both engines use it in the MoE chart above: it is lossy. We measure logit relative-L2 error of ≈3–6%, and greedy decode can diverge from the bf16 reference after ≈21 tokens. It ships opt-in for that reason; every non-KV configuration above passes strict token-identity.

What isn't tuned yet

Two paths have not yet been tuned: prefill and multi-user batching. Kernel selection in plow today is hand-placed; the measurement-calibrated auto-tuner that selects and calibrates kernels per shape has not landed yet, and those two paths are first in its queue. Decode was optimized first because batch-1 decode latency is the operative metric for interactive and voice workloads. A measured kernel-level change already improves batched decode by 34% at batch 16; when the tuner lands, prefill and throughput numbers will be published under the same protocol, before and after.

The decode result follows from the execution model: the whole forward pass is one compiled plan run by one persistent kernel, so serve-time overhead is small enough to measure precisely — +0.9% over the raw kernel on 12B, below measurement noise on 26B and 31B. The mechanics are described in the plow page.

A roofline sanity check

Batch-1 decode has a property that makes it unusually auditable: it is memory-bandwidth-bound. Every generated token must stream the model's weights — plus the growing KV cache — through the GPU once, and the arithmetic per byte is far too low to hide behind compute. That gives a hard floor any reader can check: ms/token ≥ bytes touched per token ÷ memory bandwidth. No scheduler, kernel, or compiler gets below it.

Applying the check to our own headline number: Gemma-4-31B at w8a8 is ≈31 GB of weights; at this GPU's ≈1.8 TB/s of memory bandwidth, the weight-streaming floor alone is ≈17.3 ms/token. The measured 26.35 ms at 1k context is ≈1.5× that floor; the remainder is KV reads, attention, and the non-GEMM tail, and represents unrealized headroom.

The fp8 result is consistent with the same model: bf16 doubles the weight bytes, so the weight-bound term doubles with it. The −41% end-to-end reduction against vLLM bf16 tracks the bandwidth model's prediction for halved weight traffic (measured GEMM-level reduction: −48%; the end-to-end figure is smaller because weights are not the entire byte budget). Context scaling follows the same arithmetic: per-token KV bytes grow linearly with depth, which is why the fp8-KV columns in the table separate from their bf16 counterparts as context grows — in both engines.

The same decomposition applies to any decode benchmark, this one included: convert the claim to bytes over bandwidth and account for the remainder. A number that cannot be decomposed that way warrants scrutiny. The same engine's AMD results, analyzed the same way, are in the MI350X post.