Half the slope: long-context decode on 8× AMD MI350X
Native, first-class AMD support — and where the fixed plan pays off
Gemma-4-31B decode across TP1–TP8 on 8× MI350X: from 1k to 128k context vLLM's per-token latency grows up to 2.9×, plow's grows 1.52×. The flat curve wins long context — crossovers and the not-yet-tuned paths included.
AMD, native and first-class
Plow treats CDNA4 as a first-class native target, not a port: the same Rust runtime, packet ABI, and compiled-plan execution model as our Blackwell results compile directly to AMD's MFMA pipeline. On this path, decode is a zero-launch megakernel — one dispatch per decode token, verified; a graph-capture pass over it finds nothing left to reclaim. Everything below was measured on that native path, under the same methodology as the NVIDIA campaign.
The sweep: Gemma-4-31B, bf16, decode-only, on an 8× MI350X (CDNA4) node, at tensor-parallel degrees 1, 4, and 8, head-to-head with vLLM at context depths from 1k to 128k.
The result: plow's latency curve is ~2× flatter
At short context, plow's constant term — still on hand-placed, pre-tuner kernels — keeps it behind the crossover. But as context grows the curves cross, because the two engines age very differently: from 1k to 128k, vLLM's per-token latency grows 1.9–2.9×; plow's grows 1.40–1.52×. In absolute terms, 128k of context costs vLLM about 12 extra ms on every token; it costs plow about 6.
Decode latency growth, 1k → 128k context
per-token latency at 128k as a multiple of 1k · lower is better · Gemma-4-31B bf16, 8× MI350X
Growth factor = per-token decode latency at 128k ÷ latency at 1k, same engine, same TP degree. Decode-only sweep, correctness-gated.
Because the slopes differ, there is a crossover context beyond which plow measures faster, at every parallelism degree tested. Above the crossover the gap widens with depth; below it, today's untuned constant term dominates — the quantity the auto-tuner is being built to claim.
| TP degree | plow growth 1k→128k | vLLM growth 1k→128k | crossover context |
|---|---|---|---|
| TP1 | 1.40× | 1.9× | ≈119k tokens |
| TP4 | 1.45× | 2.6× | ≈42k tokens |
| TP8 | 1.52× | 2.9× | ≈81k tokens |
Workloads dominated by long contexts — long documents, agent traces, extended conversations — sit on the favorable side of those crossovers, where the smaller slope translates directly into lower per-token latency.
Why the curve is flat: a roofline read
A roofline model gives every kernel two ceilings — peak compute and peak memory bandwidth — and says a kernel can only be as fast as the lower one at its arithmetic intensity. Decode sits far under the bandwidth ceiling: each token streams the weights and the KV cache through the chip once and does very little math per byte. That decomposes per-token latency into a useful form: a constant term (weight streaming plus fixed overheads) and a slope term that grows with context, dominated by the KV bytes scanned per token.
The chart above is a measurement of the slope term. From 1k to 128k, the extra context costs vLLM about 12 ms per token and plow about 6: plow's attention path pays roughly half as much time per unit of context. With the schedule fixed at compile time and dependencies resolved by on-device counters, the KV scan streams without an orchestration layer whose cost compounds as sequences stretch — so context growth touches only the work that genuinely scales with context.
The constant term is where the tuning work lives, and the roofline says exactly how much is on the table. Profiling the single-GPU Qwen3-4B and Llama-3.1-8B baselines, our current hand-placed kernels draw 27% of the HBM bandwidth roofline (vLLM draws 42% on the same chip — no engine is near the ceiling). The measurement-calibrated auto-tuner that selects and calibrates kernels per shape has not landed yet; that unclaimed bandwidth is what it exists to claim. Closing distance to a known ceiling is a bounded engineering task, and progress against it is directly measurable.
Caveats and scope
- The wins here are long-context decode wins. Short-context decode runs on pre-tuner kernels today; the crossovers above mark exactly where the measured advantage begins.
- Prefill on AMD hasn't been tuned either. Same story as Blackwell: it's first in the auto-tuner's queue, and its numbers will be published before-and-after.
- Baseline version skew. The AMD vLLM baseline is the vendor docker build, a different vLLM version than our NVIDIA campaign; the comparison is within-platform, and we flag the skew rather than pretend it away.
- Scope. This sweep is Gemma-4-31B bf16 decode. MoE and fp8 on AMD haven't been swept yet — when they are, the numbers will be published the same way, gated on token-identity first.
How the engine works — the packet stream, the persistent kernel, the Lean-verified rewrites — is in the plow page.