A provider prices output tokens at $15 per 1M for a Llama-70B-class model on H100. Assume the realistic production decode setup:
- H100 hourly rental: $5/hr (spot-equivalent)
- Achievable decode throughput at moderate batch (~10 concurrent users): ~150 tokens/second per H100
- Output tokens are billed at $15/M, no input tokens for this calculation
Compute:
1. Output tokens generated per H100 per hour
2. Hardware cost per million output tokens ($/M)
3. Gross margin (price - cost) per million output tokens
4. Gross margin percentage
**Unit economics** for token serving comes down to one comparison: what it costs you to make a million output tokens, versus what you sell them for. The setup gives a rental price of $5 per hour for an H100 and a decode rate of 150 tokens per second.
First, find the supply per hour. At 150 tokens per second, one hour of 3600 seconds produces 150 times 3600, which is 540,000 tokens. So a single H100 makes about 0.54 million tokens every hour it runs.
Next, turn that into cost per million. You paid $5 for that hour and got 0.54 million tokens. Dividing $5 by 0.54 gives roughly $9.26 to produce one million output tokens.
Now compare against the price. You sell at $15 per million, so your gross margin is 15 minus 9.26, about $5.74 per million. As a percentage that is 5.74 divided by 15, roughly 38 percent.
The takeaway: the margin is healthy but not huge, and the fastest way to grow it is to make each GPU emit more tokens per second, not to hunt for slightly cheaper rentals.
**Cost per token** is set by how much GPU time you rent divided by how many tokens that time produces. Start with supply. At 150 tokens per second, one hour yields 150 times 3600, which is 540,000 tokens per H100.
Cost per token follows directly:
$$\text{cost/token} = \frac{\$/\text{hr}}{\text{tok/s} \cdot 3600}$$
Plugging in gives $5 divided by 540,000, about $9.26 per million tokens. You sell at $15 per million. Gross margin is 15 minus 9.26, roughly $5.74 per million. As a fraction that is 5.74 over 15, near 38 percent.
The structural insight matters more than the digits. Cost per token scales inversely with throughput, so doubling tokens per second halves cost and roughly doubles margin at a fixed price. Rental price moves cost only linearly. That is why serving teams chase batching, speculative decoding, and quantization. Each one raises tokens per GPU-second, which compounds straight into the bottom line.
Quick reference
TL;DR
At 150 tok/s and $5/hr, an H100 makes 540k tokens/hr, costing about $9.26/M. Selling at $15/M leaves roughly $5.74/M margin, near 38%.
Think of a GPU as a vending machine you rent for $5 an hour that spits out tokens. In one hour it can drop about 540,000 tokens. So the cost of each batch of a million tokens is your hourly rent divided by how many millions you produced that hour: $5 spread across 0.54 million tokens, which works out near $9.26 per million. You then sell that same million tokens for $15. The difference, about $5.74, is what you keep before everything else like staff and networking. As a fraction of the $15 price, that gain is roughly 38 cents on the dollar. Make the machine spit out tokens faster and your rent buys more product, so the kept fraction grows.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Workloads dominated by long input prompts rather than generated output This calculation prices output decode only. Heavy prefill changes the cost basis and the dominant bottleneck entirely.
Reserved or committed-capacity contracts rather than spot rentals The hourly rate assumed here is spot equivalent. Reserved pricing and idle commitments shift the cost per token materially.
Tiny low-traffic deployments that cannot fill a batch Achievable throughput collapses far below 150 tokens per second at low concurrency, so the assumed rate does not hold.
Full business margin or pricing decisions This is gross hardware margin only. Networking, staffing, retries, and prefill all erode the real figure substantially.
Quick scan · 2–4 min
Everything important, quickly.
4 min: tokens per hour via 3600 + cost per token as rental over throughput + margin and margin percent + sensitivity to throughput versus rental + what real costs this omits.
Real-world examples
Real products, models, and research that use this idea.
1Together AI and Fireworks publish per-million token prices for Llama 4 class models that imply gross hardware margins in the 30 to 50 percent band.
2vLLM continuous batching routinely lifts decode tokens per GPU-second several fold over naive serving, directly compressing cost per token.
3DeepSeek V4 pairs Multi-head Latent Attention with fp8 KV cache to raise effective throughput per H100, improving the same margin arithmetic.
4NVIDIA H100 and B200 spot rentals on Modal, RunPod, and Lambda commonly sit near $2 to $6 per hour, the input to this cost-per-token calculation.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does doubling decode throughput change the margin at the same $15 price?
Sign in or go Pro to see the approach for this follow-up.
QWhy is throughput a stronger margin lever than GPU rental price?
Sign in or go Pro to see the approach for this follow-up.
QWhat real costs does this gross hardware margin omit?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating GPU rental price as the main margin lever. The dominant knob is tokens per second per GPU; doubling throughput roughly doubles margin at the same selling price.