Predict the hardware cost vs gross margin for serving $15/M output tokens on H100
A provider prices output tokens at $15 per 1M for a Llama-70B class model on H100. Assume the realistic production decode setup: - H100 hourly rental: $5/hr (spot equivalent) - Achievable decode throughput at moderate batch (~10 concurrent users): ~150 tokens/second per H100 - Output tokens are billed at $15/M, no input tokens for this calculation Compute: 1. Output tokens generated per H100 per hour 2. Hardware cost per million output tokens ($/M) 3. Gross margin (price - cost) per million output tokens 4. Gross margin percentage
At 150 tok/s and $5/hr, an H100 makes 540k tokens/hr, costing about $9.26/M. Selling at $15/M leaves roughly $5.74/M margin, near 38%.
Think of a GPU as a vending machine you rent for $5 an hour that spits out tokens. In one hour it can drop about 540,000 tokens. So the cost of each batch of a million tokens is your hourly rent divided by how many millions you produced that hour: $5 spread across 0.54 million tokens, which works out near $9.26 per million. You then sell that same million tokens for $15. The difference, about $5.74, is what you keep before everything else like staff and networking. As a fraction of the $15 price, that gain is roughly 38 cents on the dollar. Make the machine spit out tokens faster and your rent buys more product, so the kept fraction grows.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Cost per token is the single most important number in LLM serving economics, and this question tests whether you can compute it from first principles and then reason about which inputs actually move it. The arithmetic is deliberately simple. The judgment, knowing which lever matters, is what separates a senior answer from a junior one.
The scenario fixes three numbers: an H100 rents at $5 per hour, decode runs at about 150 tokens per second under moderate batch, and output sells at $15 per million tokens. From those three you derive supply per hour, cost per million, absolute margin, and margin percentage. Then the real work begins, which is understanding the shape of the cost function and where the sensitivity lives.
Why does an interviewer reach for this question at the senior and staff level? Because it sits at the seam between systems engineering and the business. A candidate who can only recite that batching helps throughput has half the picture. A candidate who can convert a throughput gain into a margin point, and then explain why that point is worth a quarter of headcount, is reasoning the way a serving team actually reasons. The numbers are a proxy for that judgment.
This deep dive walks the calculation step by step, promotes the cost-per-token relationship to a clean formula, runs the sensitivity analysis that reveals why throughput dominates, and then lists the real-world costs this gross figure quietly ignores. By the end you should be able to do this math on a whiteboard and immediately tell an interviewer which optimization is worth pursuing.
Step one: tokens produced per GPU-hour
Everything starts with supply. A GPU billed by the hour produces a fixed stream of tokens, so the first job is to convert a per-second rate into a per-hour quantity.
The decode rate is 150 tokens per second. An hour holds 3600 seconds. Multiplying gives 150 times 3600, which equals 540,000 tokens per hour, or 0.54 million tokens per H100-hour.
The most common slip here is using 60 instead of 3600, as if converting minutes rather than seconds. That single error shifts every downstream number by a factor of 60 and produces a nonsensical cost. Always anchor on seconds per hour, because the throughput figure is quoted per second.
It is also worth pausing on what the 150 figure represents. This is decode throughput at moderate batch, not the peak number you would read off a kernel benchmark. The peak prefill rate for the same GPU can be ten to a hundred times higher, because prefill processes the whole prompt in parallel and is compute bound. Decode is sequential and memory bandwidth bound, so its tokens per second is far lower and is the rate that actually governs output billing. Using a prefill benchmark number here would inflate supply and badly understate cost.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Together AI and Fireworks publish per-million token prices for Llama 4 class models that imply gross hardware margins in the 30 to 50 percent band.
- vLLM continuous batching routinely lifts decode tokens per GPU-second several fold over naive serving, directly compressing cost per token.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does doubling decode throughput change the margin at the same $15 price?
Cost per token is rental over throughput, so doubling tokens per second halves cost from $9.26 to about $4.63. Margin jumps from $5.74 to about $10.37, lifting gross margin from 38 percent past 69 percent. Throughput is the dominant lever.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating GPU rental price as the main margin lever. The dominant knob is tokens per second per GPU; doubling throughput roughly doubles margin at the same selling price.
60 second bullets to scan on the way to the call.
Converting tokens per second to tokens per hour with the 3600 factor
The cost per token formula as rental divided by throughput
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.