Sketch a Llama 3.1 70B self-host versus hosted-API break-even on the back of an envelope

Short answer·Medium·4.0 · 0·~3 min·Relevant atAnthropic Databricks Modal Labs

Attempt it

Run a back-of-envelope calculation for Llama 3.1 70B self-hosted on an H100 spot instance vs a hosted Llama API at $0.50/M output tokens. Assume a spot H100 at $2.5/hr, sustained ~600 output tokens/sec batched in FP8, and 30% realistic utilization across a 24-hour day. Derive the self-host per-million-tokens cost and the tokens-per-day break-even.

Free · 2 AI evals / day

**Start with the fixed bill.** An H100 spot at 2.50 dollars per hour costs 60 dollars per day if you keep it running 24/7. **Compute peak capacity.** At 600 tokens per second of sustained output (FP8 with continuous batching), the GPU can produce 600 * 86,400 equals 52 million tokens per day at full throttle. **Apply the utilization haircut.** Real services do not sit at 100% all day. A typical traffic curve sees 20-30% sustained utilization across 24 hours, because nights and off-peak hours are quiet. At 30%, realized output is about 15.5 million tokens per day. **Derive per-million cost.** 60 dollars divided by 15.5 million tokens equals 3.87 dollars per million output tokens. **Compare.** The hosted Llama API runs around 0.50 dollars per million tokens. Self-host at 30% utilization costs roughly 8x more than the API. **The break-even.** To match the API price, you need to push realized output to roughly 120 million tokens per day on the same GPU, which means sustained utilization beyond what a single H100 can deliver. Multi-GPU clusters at high steady-state batch can get there; small teams with bursty traffic cannot.

**The fixed cost** is 60 dollars per day for an H100 spot instance at 2.50 dollars per hour. That number is set the moment you provision the GPU and does not flex with traffic. **Peak throughput** at 600 output tokens per second sustained in FP8 with continuous batching is 600 * 86,400, which is 52 million tokens per day at 100% utilization. **Realized throughput at 30% utilization** (a reasonable default for a service with diurnal traffic) is 15.5 million tokens per day. At 70-80% (a saturated batch server with steady load), it rises to 36-42 million. **Per-million-token cost** is daily-bill divided by daily-tokens. At 30% utilization, that is 60 / 15.5 equals 3.87 dollars per million. At 75%, it drops to 60 / 39 equals 1.54 dollars per million. Both are above the hosted API at 0.50 dollars per million. **The break-even calculation** asks where per-million equals the API price. Setting 60 / X equals 0.50 gives X equals 120 million tokens per day. That is roughly 2.3x the peak capacity of a single H100 at 600 tokens per second. A single H100 cannot get there. The break-even point requires either multi-GPU horizontal scaling at very high sustained utilization or moving to higher-throughput hardware like H200 or B200 where per-token cost drops. **The honest reading.** For traffic up to a few tens of millions of tokens per day, the hosted API is meaningfully cheaper. Above 30-50 million tokens per day on saturated multi-GPU clusters with FP8 continuous batching, self-host starts to win, and the gap widens fast at very high sustained volume. Ops and engineering costs add roughly 50-100% on top of the GPU bill for small teams, pushing the break-even higher still.

Quick reference

TL;DR

At 30% utilization, self-host runs about 3.90 dollars per million tokens against a 0.50 dollar API price. Hosted wins until sustained throughput approaches 120M tokens/day per GPU.

Memory aid

Easy to grasp

Imagine renting a pizza oven that costs 60 dollars a day. The oven can bake 52 pizzas in a 24-hour stretch if it never stops. In reality you only get orders for one-third of the day, so you bake 15 pizzas. Your cost per pizza is 60 divided by 15, or about 4 dollars. A nearby delivery service sells the same pizza for 50 cents. Until you can keep that oven much busier or share it across many shops, the delivery service is cheaper. The same math runs LLM serving. The GPU rent is fixed by the hour. Your cost per million tokens depends entirely on how busy you keep it, and most teams keep it less busy than they think.

Key concepts

Self-hosting LLMs Pricing math NVIDIA H100

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: walk through the four steps (daily bill, peak capacity, utilization haircut, per-million derivation), compute the break-even at the stated assumptions, name the utilization tiers, and close with the operational realities that push the break-even higher for small teams.

Real-world examples

Real products, models, and research that use this idea.

Together AI and Fireworks price hosted Llama 3.1 70B at 0.50-0.90 dollars per million output tokens, undercutting most small-team self-host math.
Modal and Replicate offer H100 serverless at premium prices but pass utilization risk to the platform.
Anyscale and Baseten price dedicated H100 endpoints aimed at teams that have crossed the break-even and need predictable capacity.
Internal teams at OpenAI and Anthropic self-host because their volume is in the trillions of tokens per day where DIY economics dominate.
Snowflake Cortex prices Llama serving at near-cost for enterprise accounts as a platform attach motion.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does H200 or B200 hardware change the break-even calculation?

QWhy does input token cost matter when the question focuses on output tokens?

QWhat does spot instance interruption do to the math?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Using peak token rate as if the GPU ran flat-out 24/7. Real utilization is 20-30% by default, and that single assumption changes the break-even by an order of magnitude.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.