NVIDIA H100 SXM for LLM inference
Most of our builds and comparisons run here. The comparisons below cover Gemma-4 26B, Gemma-4 E4B and Qwen3.8 27B. Results depend on the model, sampling settings and load; vLLM leads on Qwen3.8 27B throughput.
Specs
- Architecture
- Hopper
- Memory
- 80 GB
- Bandwidth
- 3.35 TB/s
- BF16
- 1,979 TFLOPS with sparsity
- FP8
- 3,958 TFLOPS with sparsity
- FP32
- 67 TFLOPS
- Power
- Up to 700 W
Peak figures as the vendor states them, from the NVIDIA product page (checked 3 Nov 2026).
What fits
Weights take about 2 bytes a parameter in BF16 and 1 in FP8. 80 GB holds the weights of a model up to about 40B parameters in BF16, or 80B in FP8, on one card. The KV cache for each live request needs room on top, so long contexts and many requests need headroom.
Measured on the H100
Showing all results for throughput, all GPUs.
How to read the charts
plow wins shows workloads where plow leads on the selected metric. All results includes ties and vLLM wins. Each bar is one workload: right of parity, plow is ahead; left, vLLM is. Throughput counts input and output tokens. Latencies are displayed as speed ratios. Bars use a logarithmic scale capped at 4x; labels show the full ratio. The table shows all metrics for the same filtered workloads.
Gemma-4 26B-A4B on 1x H100
Gemma-4 E4B on 1x H100
Qwen3.8 27B on 1x H100
No results match these filters. Choose another metric or GPU, or select All results.
Selected results as a table
| Model and GPU | Workload | Load | Throughput | First token | Per token | Latency stat |
|---|---|---|---|---|---|---|
| Gemma-4 26B-A4B, 1x H100 | Chat mix | 1.8 sessions/s | 1.00x | 0.68x | 0.55x | p99 |
| Gemma-4 26B-A4B, 1x H100 | Agent sessions, 16k context | 64 sessions | 1.40x | 0.84x | 0.60x | p99 |
| Gemma-4 E4B, 1x H100 | 1,000 / 128 tokens | 32 requests | 1.11x | 0.74x | 0.93x | p99 |
| Qwen3.8 27B, 1x H100 | 1,000 / 128 tokens | 1 request | 0.93x | 0.48x | 1.12x | p99 |
| Qwen3.8 27B, 1x H100 | 1,000 / 128 tokens | 8 requests | 0.85x | 1.01x | 1.40x | p99 |
| Qwen3.8 27B, 1x H100 | 1,000 / 128 tokens | 32 requests | 0.96x | 0.99x | 1.25x | p99 |
| Qwen3.8 27B, 1x H100 | 1,000 / 128 tokens | 64 requests | 0.88x | 1.70x | 0.94x | p99 |
| Qwen3.8 27B, 1x H100 | 1,000 / 128 tokens | 128 requests | 0.91x | 1.22x | 0.77x | p99 |
Ratios are plow divided by vLLM, as published. For throughput higher is better; for the two latency columns lower is better.
How it compares
| GPU | Memory | Bandwidth | BF16 | FP8 | Power |
|---|---|---|---|---|---|
| H100 | 80 GB | 3.35 TB/s | 1,979 TFLOPS with sparsity | 3,958 TFLOPS with sparsity | Up to 700 W |
| MI300X | 192 GB | 5.3 TB/s | 1.3 PFLOPS (2.61 with sparsity) | 2.61 PFLOPS (5.22 with sparsity) | 750 W |
| MI325X | 256 GB | 6 TB/s | 1.3 PFLOPS (2.61 with sparsity) | 2.61 PFLOPS (5.22 with sparsity) | 1,000 W |
| MI350X | 288 GB | 8 TB/s | 2.3 PFLOPS (4.6 with sparsity) | 4.6 PFLOPS (9.2 with sparsity) | 1,000 W, passive cooling |
| MI355X | 288 GB | 8 TB/s | 2.5 PFLOPS (5 with sparsity) | 5 PFLOPS (10.1 with sparsity) | 1,400 W, passive or active cooling |
Vendors quote some peaks only with structured sparsity, marked where they do. Compare measured results, not peaks: see all benchmarks.
Questions
- Is the H100 good for LLM inference?
- Most of our builds and comparisons run here. The comparisons below cover Gemma-4 26B, Gemma-4 E4B and Qwen3.8 27B. Results depend on the model, sampling settings and load; vLLM leads on Qwen3.8 27B throughput.
- How much memory does the H100 have?
- 80 GB at 3.35 TB/s. That holds the weights of a model up to about 40B parameters in BF16, or 80B in FP8, before the KV cache.
- Which inference engines run on the H100?
- vLLM, SGLang, TensorRT-LLM and plow.
- Does plow have a tuned build for the H100?
- Yes: Gemma 4 31B, Gemma 4 E4B, Orpheus 3B, Qwen3.8 27B, Qwen3 ASR 0.6B, Veena.
Run a model on the H100
Tuned builds in the registry: Gemma 4 31B, Gemma 4 E4B, Orpheus 3B, Qwen3.8 27B, Qwen3 ASR 0.6B, Veena.