Estimate the GPU nodes needed to self-host a 70B model at 24,000 tokens per second
You are sizing self-hosted capacity for a 70B model. Load tests show one 8-GPU H100 node running your serving stack sustains 6,000 output tokens per second at your latency SLO. Peak demand is forecast at 24,000 output tokens per second. How many nodes (and total GPUs) do you need to serve peak, and how many with N+1 redundancy?
Four nodes (32 H100s) carry the peak; five nodes (40 H100s) preserve the SLO when one node is lost. Always size from your own latency-SLO load test, not vendor peak numbers.
Imagine each kitchen station can serve six thousand meals an hour at the speed your customers expect. Lunch peak demands twenty-four thousand. You need four stations to keep up. But if one cook calls in sick or you need to clean a station, you are stuck. So you set up a fifth station as a spare. That is exactly the math for GPU nodes serving a large model: divide peak demand by the measured per-station capacity, then add one full station for redundancy. The trick is in choosing the per-station number. Marketing slides quote the maximum throughput the hardware can ever hit. You need the number from your own load test at the latency your customers tolerate, which is always lower.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Capacity sizing for self-hosted LLM inference looks like a one-line division and usually is, on paper. The interesting questions are what number you divide by, what the redundancy term protects you against, and what happens above forecast peak.
This question pins all three. The per-node throughput is given as 6,000 output tokens per second at the SLO, which already assumes the right kind of load test was run. The peak is 24,000. The redundancy posture is N+1. The arithmetic gives 4 base nodes and 5 with redundancy, 32 and 40 GPUs respectively.
The deep dive walks the four hidden assumptions, because in a real planning meeting all four come up and need defensible answers: how the 6,000 was measured, why N+1 specifically, what the bill looks like, and what to do above peak.
The arithmetic and why it is the easy part
Base capacity is ceiling of peak over per-node throughput. Numerically:
At 8 H100s per node, that is 32 GPUs for base and 40 with N+1. The ceiling matters: if peak divided by per-node throughput had been 4.1, you would still need 5 base nodes, since fractional nodes do not exist and rounding down breaks the SLO.
The arithmetic is mechanical. The interesting work is everything that fed into the two input numbers. Peak demand requires a traffic forecast with confidence intervals and seasonal patterns. Per-node throughput requires a load test against the actual serving stack at the actual latency SLO. Both numbers move with product changes (new use cases, longer prompts) and infrastructure changes (vLLM upgrades, KV cache layout). Treat them as living numbers, not constants.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM and SGLang publish per batch size throughput curves that show the latency-throughput Pareto frontier on H100 and H200
- AWS Bedrock provisioned throughput is billed per model unit precisely because the same hardware delivers different effective throughput at different latency SLOs
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat changes if peak is bursty and only sustained for ten minutes a day?
Steady-state sizing to peak overprovisions. Two reasonable options: autoscale on warm spare nodes that can join the pool in minutes, or route burst traffic to a managed endpoint priced per token. The choice depends on cost per token versus cost per hour of idle GPU.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Sizing from a vendor's peak throughput chart instead of your own load test at the latency SLO. Batching pressure degrades latency well before the GPU saturates, so the practical number is lower.
60 second bullets to scan on the way to the call.
Why per-node throughput must come from your own load test, not vendor peak benchmarks
How to find the throughput-latency Pareto frontier with your serving stack
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.