Inference benchmarks

Where plow leads on H100 and MI350X. Explore wins by metric and GPU, or choose All results to see the other measured outcomes.

Showing plow wins for throughput, all GPUs.

How to read the charts

plow wins shows workloads where plow leads on the selected metric. All results includes ties and vLLM wins. Each bar is one workload: right of parity, plow is ahead; left, vLLM is. Throughput counts input and output tokens. Latencies are displayed as speed ratios. Bars use a logarithmic scale capped at 4x; labels show the full ratio. The table shows all metrics for the same filtered workloads.

Gemma-4 26B-A4B on 1x H100

Against
vLLM 0.29
Precision
FP8
Run
Accuracy
FP32-reference gate passed
Agent sessions, 16k context64 sessions 1.40x
Chat mix1.8 sessions/s 1.47x
Agent sessions, 16k context64 sessions 1.19x
Chat mix1.8 sessions/s 1.82x
Agent sessions, 16k context64 sessions 1.67x

The chat mix arrives at a fixed rate, so throughput is pinned by the arrivals and the wins are in latency.

Latency is the 99th percentile (p99) of requests. Settings and source, every row (CSV).

Gemma-4 E4B on 1x H100

Against
vLLM 0.29
Precision
BF16
Run
Accuracy
FP32-reference and logit-parity gates passed
1,000 / 128 tokens32 requests 1.11x
1,000 / 128 tokens32 requests 1.35x
1,000 / 128 tokens32 requests 1.08x

Greedy decoding, 1,000 input tokens and 128 output tokens, two repeats at 32 concurrent requests.

Latency is the 99th percentile (p99) of requests. Settings and source, every row (CSV).

Selected results as a table
plow / vLLM ratios for the filtered workloads
Model and GPUWorkloadLoadThroughputFirst tokenPer tokenLatency stat
Gemma-4 26B-A4B, 1x H100Agent sessions, 16k context64 sessions1.40x0.84x0.60xp99
Gemma-4 E4B, 1x H1001,000 / 128 tokens32 requests1.11x0.74x0.93xp99

Ratios are plow divided by vLLM, as published. For throughput higher is better; for the two latency columns lower is better.

Other measured results

Results reported in a post rather than a cell table: decode at long context, start-up time, CPU serving and day-zero bring-ups.

  • Gemma-4 31B, long-context decode, 8x AMD MI350X 1.40x to 1.52x

    How much plow’s time per token grows from 1k to 128k context, against 1.9x to 2.9x for vLLM. plow is faster past about 42k tokens at TP4 and 81k at TP8.

    , source
  • Kimi-K3, day-zero bring-up, 8x AMD MI355X 28.9 ms per token

    34.6 tokens per second at TP8 and 32k context, with all eight ranks token-identical. Not a vLLM comparison.

    , source
  • Kimi-K3 serving, 8x AMD MI325X 131 tokens per second

    Aggregate generated tokens per second at 32 concurrent requests, measured with the vLLM 0.27 client. Not a vLLM engine comparison.

    , source
  • Gemma-4 decode on Blackwell, RTX PRO 6000 41% lower decode latency

    Gemma-4 31B dense decode latency against vLLM BF16, one RTX PRO 6000, batch one, 1k to 32k context. This is a decode-only measurement.

    , source
  • Gemma-4 12B start-up, GH200 11 s vs 122 s

    Time until the API is ready, plowrt against vLLM.

    , source
  • Gemma-4 on one CPU socket, Xeon 6975P-C 3.3x to 5.8x

    Serving throughput against vLLM 0.30 CPU at one request, E2B to 26B-A4B.

    , source

Every number is checkable

  • The comparison board includes matched configurations with passed quality checks and no repeat-spread flags in the displayed metrics.
  • Each comparison links to its quality checks, build, engine versions and benchmark settings.
  • Results apply to the configuration and traffic shown. Sampling, prefix caching and workload differences matter; consult each source before extrapolating.
  • The source reports contain additional experiments. The board is a selection, not an exhaustive campaign history; standalone measurements above use their own documented methods.

Compare engines on plow vs vLLM and SGLang vs vLLM vs plow, or by GPU in the GPU guides.