AMD Instinct MI300X for LLM inference
Its 192 GB holds a 31B model’s BF16 weights with memory left for the KV cache. We do not currently publish a qualified engine comparison for this card on this site.
Specs
- Architecture
- CDNA 3, 304 compute units
- Memory
- 192 GB HBM3
- Bandwidth
- 5.3 TB/s
- BF16
- 1.3 PFLOPS (2.61 with sparsity)
- FP8
- 2.61 PFLOPS (5.22 with sparsity)
- FP32
- 163.4 TFLOPS
- Power
- 750 W
- Launched
Peak figures as the vendor states them, from the AMD product page (checked 3 Nov 2026).
What fits
Weights take about 2 bytes a parameter in BF16 and 1 in FP8. 192 GB holds the weights of a model up to about 96B parameters in BF16, or 192B in FP8, on one card. The KV cache for each live request needs room on top, so long contexts and many requests need headroom.
Measured on the MI300X
No qualified engine comparison is currently published on this site for this card.
How it compares
| GPU | Memory | Bandwidth | BF16 | FP8 | Power |
|---|---|---|---|---|---|
| H100 | 80 GB | 3.35 TB/s | 1,979 TFLOPS with sparsity | 3,958 TFLOPS with sparsity | Up to 700 W |
| MI300X | 192 GB | 5.3 TB/s | 1.3 PFLOPS (2.61 with sparsity) | 2.61 PFLOPS (5.22 with sparsity) | 750 W |
| MI325X | 256 GB | 6 TB/s | 1.3 PFLOPS (2.61 with sparsity) | 2.61 PFLOPS (5.22 with sparsity) | 1,000 W |
| MI350X | 288 GB | 8 TB/s | 2.3 PFLOPS (4.6 with sparsity) | 4.6 PFLOPS (9.2 with sparsity) | 1,000 W, passive cooling |
| MI355X | 288 GB | 8 TB/s | 2.5 PFLOPS (5 with sparsity) | 5 PFLOPS (10.1 with sparsity) | 1,400 W, passive or active cooling |
Vendors quote some peaks only with structured sparsity, marked where they do. Compare measured results, not peaks: see all benchmarks.
Questions
- Is the MI300X good for LLM inference?
- Its 192 GB holds a 31B model’s BF16 weights with memory left for the KV cache. We do not currently publish a qualified engine comparison for this card on this site.
- How much memory does the MI300X have?
- 192 GB of HBM3 at 5.3 TB/s. That holds the weights of a model up to about 96B parameters in BF16, or 192B in FP8, before the KV cache.
- Which inference engines run on the MI300X?
- vLLM, SGLang and plow, through ROCm.
- Does plow have a tuned build for the MI300X?
- No tuned build is currently listed in the public registry.