AMD Instinct MI355X for LLM inference
The higher-clocked MI350 at 1,400 W. We brought up Kimi-K3 here on day zero: 28.9 ms per token at TP8 and 32k context, with all eight GPUs token-identical. That run is not a comparison against vLLM.
Specs
- Architecture
- CDNA 4, 256 compute units
- Memory
- 288 GB HBM3E
- Bandwidth
- 8 TB/s
- BF16
- 2.5 PFLOPS (5 with sparsity)
- FP8
- 5 PFLOPS (10.1 with sparsity)
- FP4
- 10.1 PFLOPS (MXFP4)
- FP32
- 157.3 TFLOPS
- Power
- 1,400 W, passive or active cooling
- Launched
Peak figures as the vendor states them, from the AMD product page (checked 3 Nov 2026).
What fits
Weights take about 2 bytes a parameter in BF16 and 1 in FP8. 288 GB holds the weights of a model up to about 144B parameters in BF16, or 288B in FP8, on one card. The KV cache for each live request needs room on top, so long contexts and many requests need headroom.
Measured on the MI355X
- Kimi-K3, day-zero bring-up, 8x AMD MI355X 28.9 ms per token
34.6 tokens per second at TP8 and 32k context, with all eight ranks token-identical. Not a vLLM comparison.
, source
How it compares
| GPU | Memory | Bandwidth | BF16 | FP8 | Power |
|---|---|---|---|---|---|
| H100 | 80 GB | 3.35 TB/s | 1,979 TFLOPS with sparsity | 3,958 TFLOPS with sparsity | Up to 700 W |
| MI300X | 192 GB | 5.3 TB/s | 1.3 PFLOPS (2.61 with sparsity) | 2.61 PFLOPS (5.22 with sparsity) | 750 W |
| MI325X | 256 GB | 6 TB/s | 1.3 PFLOPS (2.61 with sparsity) | 2.61 PFLOPS (5.22 with sparsity) | 1,000 W |
| MI350X | 288 GB | 8 TB/s | 2.3 PFLOPS (4.6 with sparsity) | 4.6 PFLOPS (9.2 with sparsity) | 1,000 W, passive cooling |
| MI355X | 288 GB | 8 TB/s | 2.5 PFLOPS (5 with sparsity) | 5 PFLOPS (10.1 with sparsity) | 1,400 W, passive or active cooling |
Vendors quote some peaks only with structured sparsity, marked where they do. Compare measured results, not peaks: see all benchmarks.
Questions
- Is the MI355X good for LLM inference?
- The higher-clocked MI350 at 1,400 W. We brought up Kimi-K3 here on day zero: 28.9 ms per token at TP8 and 32k context, with all eight GPUs token-identical. That run is not a comparison against vLLM.
- How much memory does the MI355X have?
- 288 GB of HBM3E at 8 TB/s. That holds the weights of a model up to about 144B parameters in BF16, or 288B in FP8, before the KV cache.
- Which inference engines run on the MI355X?
- vLLM, SGLang and plow, through ROCm.
- Does plow have a tuned build for the MI355X?
- No tuned build is currently listed in the public registry.