AMD Instinct MI350X for LLM inference

CDNA 4 adds FP4 and FP6 and raises bandwidth to 8 TB/s. On eight MI350X, plow’s time per token grows 1.40 to 1.52 times from 1k to 128k context, against 1.9 to 2.9 times for vLLM. On GLM-5.3-Flash, plow’s p99 time to first token is 0.28 to 0.62 times vLLM’s in four of five cells, while vLLM leads on throughput.

Specs

Architecture
CDNA 4, 256 compute units
Memory
288 GB HBM3E
Bandwidth
8 TB/s
BF16
2.3 PFLOPS (4.6 with sparsity)
FP8
4.6 PFLOPS (9.2 with sparsity)
FP4
9.2 PFLOPS (MXFP4)
FP32
144.2 TFLOPS
Power
1,000 W, passive cooling
Launched

Peak figures as the vendor states them, from the AMD product page (checked 3 Nov 2026).

What fits

Weights take about 2 bytes a parameter in BF16 and 1 in FP8. 288 GB holds the weights of a model up to about 144B parameters in BF16, or 288B in FP8, on one card. The KV cache for each live request needs room on top, so long contexts and many requests need headroom.

Measured on the MI350X

Showing all results for throughput, all GPUs.

How to read the charts

plow wins shows workloads where plow leads on the selected metric. All results includes ties and vLLM wins. Each bar is one workload: right of parity, plow is ahead; left, vLLM is. Throughput counts input and output tokens. Latencies are displayed as speed ratios. Bars use a logarithmic scale capped at 4x; labels show the full ratio. The table shows all metrics for the same filtered workloads.

GLM-5.3-Flash on 8x AMD MI350X

Against
vLLM 0.31.1rc1 (ROCm)
Precision
FP8
Run
Accuracy
Accuracy gate passed
1,024 / 128 tokens8 requests vLLM 1.10x
4,096 / 128 tokens1 request vLLM 1.20x
4,096 / 128 tokens8 requests vLLM 1.04x
16,384 / 128 tokens1 request vLLM 1.22x
16,384 / 128 tokens128 requests vLLM 1.54x
1,024 / 128 tokens8 requests 2.78x
4,096 / 128 tokens1 request 3.57x
4,096 / 128 tokens8 requests 2.08x
16,384 / 128 tokens1 request 1.61x
16,384 / 128 tokens128 requests vLLM 1.53x
1,024 / 128 tokens8 requests 1.22x
4,096 / 128 tokens1 request vLLM 1.26x
4,096 / 128 tokens8 requests 1.41x
16,384 / 128 tokens1 request vLLM 1.26x
16,384 / 128 tokens128 requests 1.25x

Both engines use the same eight-GPU host and benchmark client. Prefix caching is on for plow and off for vLLM; these results apply to those settings.

Latency is the 99th percentile (p99) of requests. Settings and source, every row (CSV).

Selected results as a table
plow / vLLM ratios for the filtered workloads
Model and GPUWorkloadLoadThroughputFirst tokenPer tokenLatency stat
GLM-5.3-Flash, 8x AMD MI350X1,024 / 128 tokens8 requests0.91x0.36x0.82xp99
GLM-5.3-Flash, 8x AMD MI350X4,096 / 128 tokens1 request0.83x0.28x1.26xp99
GLM-5.3-Flash, 8x AMD MI350X4,096 / 128 tokens8 requests0.96x0.48x0.71xp99
GLM-5.3-Flash, 8x AMD MI350X16,384 / 128 tokens1 request0.82x0.62x1.26xp99
GLM-5.3-Flash, 8x AMD MI350X16,384 / 128 tokens128 requests0.65x1.53x0.80xp99

Ratios are plow divided by vLLM, as published. For throughput higher is better; for the two latency columns lower is better.

  • Gemma-4 31B, long-context decode, 8x AMD MI350X 1.40x to 1.52x

    How much plow’s time per token grows from 1k to 128k context, against 1.9x to 2.9x for vLLM. plow is faster past about 42k tokens at TP4 and 81k at TP8.

    , source

How it compares

GPUMemoryBandwidthBF16FP8Power
H10080 GB3.35 TB/s1,979 TFLOPS with sparsity3,958 TFLOPS with sparsityUp to 700 W
MI300X192 GB5.3 TB/s1.3 PFLOPS (2.61 with sparsity)2.61 PFLOPS (5.22 with sparsity)750 W
MI325X256 GB6 TB/s1.3 PFLOPS (2.61 with sparsity)2.61 PFLOPS (5.22 with sparsity)1,000 W
MI350X288 GB8 TB/s2.3 PFLOPS (4.6 with sparsity)4.6 PFLOPS (9.2 with sparsity)1,000 W, passive cooling
MI355X288 GB8 TB/s2.5 PFLOPS (5 with sparsity)5 PFLOPS (10.1 with sparsity)1,400 W, passive or active cooling

Vendors quote some peaks only with structured sparsity, marked where they do. Compare measured results, not peaks: see all benchmarks.

Questions

Is the MI350X good for LLM inference?
CDNA 4 adds FP4 and FP6 and raises bandwidth to 8 TB/s. On eight MI350X, plow’s time per token grows 1.40 to 1.52 times from 1k to 128k context, against 1.9 to 2.9 times for vLLM. On GLM-5.3-Flash, plow’s p99 time to first token is 0.28 to 0.62 times vLLM’s in four of five cells, while vLLM leads on throughput.
How much memory does the MI350X have?
288 GB of HBM3E at 8 TB/s. That holds the weights of a model up to about 144B parameters in BF16, or 288B in FP8, before the KV cache.
Which inference engines run on the MI350X?
vLLM, SGLang and plow, through ROCm.
Does plow have a tuned build for the MI350X?
No tuned build is currently listed in the public registry.