Sketch a Llama 3.1 70B self-host versus hosted API break even on the back of an envelope

Short answer·Medium·4.0 · 0·~3 min·Asked atCrewai Glean Servicenow·Relevant atAnthropic Databricks Modal Labs

Attempt it

Run a back of envelope calculation for Llama 3.1 70B self-hosted on an H100 spot instance vs a hosted Llama API at $0.50/M output tokens. Assume a spot H100 at $2.5/hr, sustained ~600 output tokens/sec batched in FP8, and 30% realistic utilization across a 24 hour day. Derive the self-host per million tokens cost and the tokens per day break even.

Free · 2 AI evals / day

**Start with the fixed bill.** An H100 spot at 2.50 dollars per hour costs 60 dollars per day if you keep it running 24/7. **Compute peak capacity.** At 600 tokens per second of sustained output (FP8 with continuous batching), the GPU can produce 600 * 86,400 equals 52 million tokens per day at full throttle. **Apply the utilization haircut.** Real services do not sit at 100% all day. A typical traffic curve sees 20-30% sustained utilization across 24 hours, because nights and off-peak hours are quiet. At 30%, realized output is about 15.5 million tokens per day. **Derive per-million cost.** 60 dollars divided by 15.5 million tokens equals 3.87 dollars per million output tokens. **Compare.** The hosted Llama API runs around 0.50 dollars per million tokens. Self-host at 30% utilization costs roughly 8x more than the API. **The break-even.** To match the API price, you need to push realized output to roughly 120 million tokens per day on the same GPU, which means sustained utilization beyond what a single H100 can deliver. Multi-GPU clusters at high steady-state batch can get there; small teams with bursty traffic cannot.

**The fixed cost** is 60 dollars per day for an H100 spot instance at 2.50 dollars per hour. That number is set the moment you provision the GPU and does not flex with traffic. **Peak throughput** at 600 output tokens per second sustained in FP8 with continuous batching is 600 * 86,400, which is 52 million tokens per day at 100% utilization. **Realized throughput at 30% utilization** (a reasonable default for a service with diurnal traffic) is 15.5 million tokens per day. At 70-80% (a saturated batch server with steady load), it rises to 36-42 million. **Per-million-token cost** is daily-bill divided by daily-tokens. At 30% utilization, that is 60 / 15.5 equals 3.87 dollars per million. At 75%, it drops to 60 / 39 equals 1.54 dollars per million. Both are above the hosted API at 0.50 dollars per million. **The break-even calculation** asks where per-million equals the API price. Setting 60 / X equals 0.50 gives X equals 120 million tokens per day. That is roughly 2.3x the peak capacity of a single H100 at 600 tokens per second. A single H100 cannot get there. The break-even point requires either multi-GPU horizontal scaling at very high sustained utilization or moving to higher-throughput hardware like H200 or B200 where per-token cost drops. **The honest reading.** For traffic up to a few tens of millions of tokens per day, the hosted API is meaningfully cheaper. Above 30-50 million tokens per day on saturated multi-GPU clusters with FP8 continuous batching, self-host starts to win, and the gap widens fast at very high sustained volume. Ops and engineering costs add roughly 50-100% on top of the GPU bill for small teams, pushing the break-even higher still.

TL;DR

At 30% utilization, self-host runs about 3.90 dollars per million tokens against a 0.50 dollar API price. Hosted wins until sustained throughput approaches 120M tokens/day per GPU.

Memory aid

Easy to grasp

Imagine renting a pizza oven that costs 60 dollars a day. The oven can bake 52 pizzas in a 24-hour stretch if it never stops. In reality you only get orders for one-third of the day, so you bake 15 pizzas. Your cost per pizza is 60 divided by 15, or about 4 dollars. A nearby delivery service sells the same pizza for 50 cents. Until you can keep that oven much busier or share it across many shops, the delivery service is cheaper. The same math runs LLM serving. The GPU rent is fixed by the hour. Your cost per million tokens depends entirely on how busy you keep it, and most teams keep it less busy than they think.

Key concepts

Self-hosting LLMs Pricing math NVIDIA H100

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Self-host versus hosted API is one of the most-asked questions in production LLM serving, and the answer is almost always a back-of-envelope calculation rather than a deep architectural argument. Three numbers carry the math: GPU cost per hour, achievable tokens per second, and realistic sustained utilization. Plug them into per-million-token cost and compare against the hosted price.

The twist is that one of those three numbers, utilization, swings by 5x across realistic deployments. Naive serverless services land at 15-25%. Well-tuned batch services reach 50-70%. Saturated offline-batch workloads approach 90%+. The same hardware can sit on either side of the break-even depending on which tier the workload actually achieves.

This deep dive walks through the calculation step by step using the numbers in the prompt, computes per-million cost across utilization tiers, derives the break-even point, and closes with the operational realities that push the break-even higher for most teams. By the end you should be able to land this calculation in a five-minute whiteboard session and give a defensible answer to whether a given workload should be self-hosted.

The fixed bill: GPU hourly cost times 24

An H100 spot instance at 2.50 dollars per hour costs 60 dollars per day if you keep it running for 24 hours. Reserved capacity removes the spot interruption risk but pushes the price closer to 3.50-4.00 dollars per hour, yielding 84-96 dollars per day. On-demand H100 SXM at neocloud providers (Lambda, CoreWeave, Together AI) lands in the same on-demand range.

The fixed bill is the most important property of self-hosting: you pay it whether or not requests arrive. A GPU sitting idle costs the same as a saturated GPU. This is the opposite of the hosted API model, which charges per token consumed.

For planning purposes, use 60 dollars per day per H100 spot as the baseline. Scale up for on-demand pricing (1.4-1.6x), multi-GPU setups (linear with GPU count), or H200/B200 hardware (1.3-2.0x per GPU at higher throughput).

Peak throughput: 600 tokens per second sustained

Utilization: the swing variable

Per-million cost derivation and the API comparison

The break-even point

When to self-host and when to use the API

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Together AI and Fireworks price hosted Llama 3.1 70B at 0.50-0.90 dollars per million output tokens, undercutting most small-team self-host math.
Modal and Replicate offer H100 serverless at premium prices but pass utilization risk to the platform.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does H200 or B200 hardware change the break-even calculation?

H200 has 40% more decode throughput than H100 at modest price premium. B200 roughly doubles H100 throughput. At similar utilization, per-million cost drops proportionally. B200 brings single-GPU per-million cost down toward 0.40-0.60 dollars at high utilization, which can match hosted API prices for teams that can sustain the load.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Using peak token rate as if the GPU ran flat-out 24/7. Real utilization is 20-30% by default, and that single assumption changes the break-even by an order of magnitude.

60 second bullets to scan on the way to the call.

Compute daily GPU cost from hourly rate
Compute peak tokens per day at 100% utilization