Living guide · Last updated Oct 12, 2026
Which GPU should you rent for LLM inference?
The short version
- Size the memory first. Token generation is memory-bound, so capacity and bandwidth matter more than peak FLOPS.
- Fit on one GPU if you can. It removes tensor parallelism, synchronization, and most failure modes. The 256 to 288 GB GPUs (MI325X, B300, MI355X) fit much more on one device.
- If you need several GPUs on one model, pay for the interconnect. NVLink nodes scale better than PCIe cards.
- Compare the bill, not the hourly rate. Minimum node sizes, egress, preemption, and idle time change the real cost.
Step 1: How many GPUs does your model need?
Weights take about 2 bytes per parameter at FP16 and about 1 at FP8. On top of that you need room for the KV cache, which grows with batch size and context length. I add 20% headroom as a starting point. Long contexts or big batches need more.
Counts round up to 1, 2, 4 or 8, because tensor parallelism splits a model evenly across GPUs. Above 8 means more than one 8-GPU node.
| GPU | 8B19 GB | 32B77 GB | 70B168 GB | 123B295 GB | 405B972 GB |
|---|---|---|---|---|---|
| H10080 GB | 1 | 1 | 4 | 4 | 162 nodes |
| H200141 GB | 1 | 1 | 2 | 4 | 8 |
| B200180 GB | 1 | 1 | 1 | 2 | 8 |
| B300288 GB | 1 | 1 | 1 | 2 | 4 |
| MI300X192 GB | 1 | 1 | 1 | 2 | 8 |
| MI325X256 GB | 1 | 1 | 1 | 2 | 4 |
| MI355X288 GB | 1 | 1 | 1 | 2 | 4 |
| GPU | 8B10 GB | 32B38 GB | 70B84 GB | 123B148 GB | 405B486 GB |
|---|---|---|---|---|---|
| H10080 GB | 1 | 1 | 2 | 2 | 8 |
| H200141 GB | 1 | 1 | 1 | 2 | 4 |
| B200180 GB | 1 | 1 | 1 | 1 | 4 |
| B300288 GB | 1 | 1 | 1 | 1 | 2 |
| MI300X192 GB | 1 | 1 | 1 | 1 | 4 |
| MI325X256 GB | 1 | 1 | 1 | 1 | 2 |
| MI355X288 GB | 1 | 1 | 1 | 1 | 2 |
Try other sizes, including INT4, in the will-it-fit calculator.
Step 2: Compare the specs that matter for inference
| GPU | Memory | Bandwidth | GPU to GPU | Software |
|---|---|---|---|---|
| NVIDIA H100 | 80 GB HBM3 | 3.35 TB/s | NVLink 4, 900 GB/s | CUDA |
| NVIDIA H200 | 141 GB HBM3e | 4.8 TB/s | NVLink 4, 900 GB/s | CUDA |
| NVIDIA B200 | 180 GB HBM3e | 8 TB/s | NVLink 5, 1.8 TB/s | CUDA |
| NVIDIA B300 | 288 GB HBM3e | 8 TB/s | NVLink 5, 1.8 TB/s | CUDA |
| AMD MI300X | 192 GB HBM3 | 5.3 TB/s | Infinity Fabric | ROCm |
| AMD MI325X | 256 GB HBM3e | 6 TB/s | Infinity Fabric | ROCm |
| AMD MI355X | 288 GB HBM3e | 8 TB/s | Infinity Fabric | ROCm |
Because decode is memory-bound, bandwidth sets the ceiling on tokens per second per GPU. The H200 has 43% more bandwidth than the H100 on the same compute die. B200, B300, and MI355X all reach 8 TB/s, about 2.4 times an H100.
Step 3: One GPU, or several working together?
- Each GPU runs its own copy of the model: PCIe cards are fine. Save the money.
- Several GPUs share one model (tensor parallelism): every layer needs an all-reduce for every token, so the GPU-to-GPU link becomes the bottleneck. NVLink 4 moves 900 GB/s per GPU and NVLink 5 doubles that to 1.8 TB/s, versus about 128 GB/s for PCIe Gen5 x16. Buy SXM nodes.
- Scaling to 8 GPUs: in published 8-GPU training comparisons of H200 and MI300X, NVLink nodes reached around 87% scaling efficiency versus roughly 78% for MI300X.
Step 4: Check the price, then the real bill
Lowest listed on-demand price per GPU-hour, sorted by price per GB of memory:
| GPU | Per hour | Per GB per hour |
|---|---|---|
| AMD MI325X | $2.25 | $0.009 |
| AMD MI300X | $2.39 | $0.012 |
| NVIDIA H200 | $2.09 | $0.015 |
| NVIDIA B300 | $4.99 | $0.017 |
| AMD MI355X | $5.99 | $0.021 |
| NVIDIA B200 | $3.75 | $0.021 |
| NVIDIA H100 | $1.71 | $0.021 |
Before you commit, check four things that change the real cost:
- Minimum node size. Some providers only sell full 8-GPU nodes.
- Egress. Moving weights and data out can erase the savings.
- Preemption. Cheap is often spot, which is bad for anything user-facing.
- Utilization. A GPU you rent and don't keep busy is the most expensive one.
My picks
| If you are… | Start with |
|---|---|
| Serving a small model, 32B or under at FP8 | H100, the cheapest per hour |
| Fitting the most model per dollar of memory | MI325X, then MI300X |
| Fitting the biggest model on one GPU | B300 or MI355X, 288 GB |
| Chasing the highest tokens per second per GPU | B200, B300 or MI355X, 8 TB/s |
| Running tensor parallelism across 4 or 8 GPUs | B200 or B300 on NVLink 5 |
| Depending on CUDA-only kernels | Any NVIDIA option |
For a deeper comparison of two of these, read H200 vs MI300X: what you're actually paying for.
Frequently asked questions
How much GPU memory does a 70B model need?
About 140 GB for the weights at FP16, or about 70 GB at FP8. Add room for the KV cache: with 20% headroom that is about 168 GB at FP16 and 84 GB at FP8. Long contexts and large batches need more.
How many GPUs do I need to run a 70B model?
At FP16 (about 168 GB with headroom): 4 NVIDIA H100, 2 NVIDIA H200, 1 NVIDIA B200, 1 NVIDIA B300, 1 AMD MI300X, 1 AMD MI325X, 1 AMD MI355X. At FP8 (about 84 GB): 2 NVIDIA H100, 1 NVIDIA H200, 1 NVIDIA B200, 1 NVIDIA B300, 1 AMD MI300X, 1 AMD MI325X, 1 AMD MI355X. Counts round up to 1, 2, 4 or 8 for tensor parallelism.
How many GPUs do I need to run a 405B model?
At FP8 (about 486 GB with headroom): 8 NVIDIA H100, 4 NVIDIA H200, 4 NVIDIA B200, 2 NVIDIA B300, 4 AMD MI300X, 2 AMD MI325X, 2 AMD MI355X. At FP16 (about 972 GB): 16 NVIDIA H100, 8 NVIDIA H200, 8 NVIDIA B200, 4 NVIDIA B300, 8 AMD MI300X, 4 AMD MI325X, 4 AMD MI355X. Above 8 GPUs means more than one 8-GPU node.
Which GPU is cheapest for LLM inference?
By hourly rate, NVIDIA H100 at about $1.71 per GPU-hour. By cost per GB of memory, AMD MI325X at about $0.009 per GB-hour. Prices are the lowest listed on-demand rates on getdeploying.com, Oct 12, 2026.
What is the difference between the H100 and the H200?
They use the same compute die. The H200 has 141 GB of HBM3e at 4.8 TB/s versus 80 GB of HBM3 at 3.35 TB/s on the H100, so it holds bigger models and generates tokens faster, while prompt processing speed is about the same.
What is the difference between the B200 and the B300?
Both are NVIDIA Blackwell GPUs with 8 TB/s of memory bandwidth and NVLink 5 at 1.8 TB/s. The B300 has 288 GB of HBM3e versus 180 GB on the B200, so it fits larger models on fewer GPUs.
Changelog
- Oct 12, 2026: First version. Covers H100, H200, B200, B300, MI300X, MI325X, and MI355X, with prices from getdeploying.com.
Figures are estimates for planning, not quotes. If a number here is wrong or has gone stale, tell me and I'll fix it.