AMD Instinct MI325X for LLM inference
The same CDNA 3 compute as the MI300X, with 256 GB of HBM3E at 6 TB/s. We serve the full 93-layer Kimi-K3 on eight MI325X at 131 generated tokens per second across 32 concurrent requests; that run is not a comparison against vLLM.
Specs
- Architecture
- CDNA 3, 304 compute units
- Memory
- 256 GB HBM3E
- Bandwidth
- 6 TB/s
- BF16
- 1.3 PFLOPS (2.61 with sparsity)
- FP8
- 2.61 PFLOPS (5.22 with sparsity)
- FP32
- 163.4 TFLOPS
- Power
- 1,000 W
- Launched
Peak figures as the vendor states them, from the AMD product page (checked 3 Nov 2026).
What fits
Weights take about 2 bytes a parameter in BF16 and 1 in FP8. 256 GB holds the weights of a model up to about 128B parameters in BF16, or 256B in FP8, on one card. The KV cache for each live request needs room on top, so long contexts and many requests need headroom.
Measured on the MI325X
- Kimi-K3 serving, 8x AMD MI325X 131 tokens per second
Aggregate generated tokens per second at 32 concurrent requests, measured with the vLLM 0.27 client. Not a vLLM engine comparison.
, source
How it compares
| GPU | Memory | Bandwidth | BF16 | FP8 | Power |
|---|---|---|---|---|---|
| H100 | 80 GB | 3.35 TB/s | 1,979 TFLOPS with sparsity | 3,958 TFLOPS with sparsity | Up to 700 W |
| MI300X | 192 GB | 5.3 TB/s | 1.3 PFLOPS (2.61 with sparsity) | 2.61 PFLOPS (5.22 with sparsity) | 750 W |
| MI325X | 256 GB | 6 TB/s | 1.3 PFLOPS (2.61 with sparsity) | 2.61 PFLOPS (5.22 with sparsity) | 1,000 W |
| MI350X | 288 GB | 8 TB/s | 2.3 PFLOPS (4.6 with sparsity) | 4.6 PFLOPS (9.2 with sparsity) | 1,000 W, passive cooling |
| MI355X | 288 GB | 8 TB/s | 2.5 PFLOPS (5 with sparsity) | 5 PFLOPS (10.1 with sparsity) | 1,400 W, passive or active cooling |
Vendors quote some peaks only with structured sparsity, marked where they do. Compare measured results, not peaks: see all benchmarks.
Questions
- Is the MI325X good for LLM inference?
- The same CDNA 3 compute as the MI300X, with 256 GB of HBM3E at 6 TB/s. We serve the full 93-layer Kimi-K3 on eight MI325X at 131 generated tokens per second across 32 concurrent requests; that run is not a comparison against vLLM.
- How much memory does the MI325X have?
- 256 GB of HBM3E at 6 TB/s. That holds the weights of a model up to about 128B parameters in BF16, or 256B in FP8, before the KV cache.
- Which inference engines run on the MI325X?
- vLLM, SGLang and plow, through ROCm.
- Does plow have a tuned build for the MI325X?
- No tuned build is currently listed in the public registry.