Cloud providers sell GPU capacity in two main flavours. **On-demand** instances are guaranteed for as long as you pay; spot instances cost much less but can be reclaimed by the provider at short notice when someone else wants on-demand capacity.
For H100 GPUs in 2026, spot pricing typically runs around 1.50 to 2.50 dollars per hour, compared to 3 to 5 dollars per hour on-demand depending on cloud and region. That is a real 40 to 70 percent discount.
The catch is the eviction. The provider can give you anywhere from a few seconds to a few minutes of warning, then take the GPU back. If your workload was serving a live customer in that moment, the request fails and the user sees an error.
The rule of thumb is simple. Spot is safe when (a) the job is stateless or can resume from a checkpoint, and (b) nobody is waiting in real time. Overnight batch evals, dataset labeling, offline embedding generation, and training jobs with frequent checkpoints all fit. Live chat serving, per-user pinned sessions, and launch-window traffic do not.
Option B in this question is the textbook fit: it can resume from a checkpoint and nobody is watching a screen waiting for it.
**Spot H100s** cost roughly half of on-demand, around 1.50 to 2.50 dollars per hour versus 3 to 5 dollars, in exchange for the cloud's right to preempt the instance with seconds to minutes of warning.
That discount only pays off for workloads that can absorb a preemption without anyone noticing. Two conditions matter: the job is stateless or checkpoint-resumable, and there is no live latency budget. Option B, a nightly batch eval that resumes from a checkpoint, hits both. The eviction just costs a few minutes of wasted work on the next launch.
The other three options each fail one or both checks. Option A puts the sole replica of a customer-facing chat behind a live SLO; a single preemption is an outage. Option C pins per-user LoRA state and active sessions to the GPU; eviction drops conversations mid-token. Option D commits to 99.9 percent availability during a launch window where preemption probability blows the target without a pre-warmed on-demand failover.
The broader principle: spot pricing is a batch and offline tool. The moment a workload carries either a live SLO or per-request state that cannot be re-hydrated quickly, it belongs on on-demand, reserved, or provisioned-throughput capacity. Real deployments often use both, with spot soaking up offline work and on-demand carrying the user-facing fleet.
Quick reference
TL;DR
Spot H100 capacity is roughly half on-demand price but can be preempted with seconds of notice, so it fits only stateless, latency-tolerant, checkpoint-resumable batch jobs.
Imagine a parking lot that offers half-price spots, but the manager can ask you to leave in two minutes if a full-price customer shows up. If you are running into the store for a quick errand and your car is empty, that deal is fine. If you are mid-surgery in an ambulance parked there, losing the spot is a disaster. Spot GPU pricing works the same way. Cloud providers sell unused H100 capacity at a steep discount with the right to reclaim it at any moment. Batch jobs that can pause and pick up from a saved point handle the eviction gracefully. Live customer traffic does not, because every preempted second is a user staring at a blank screen.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Customer-facing realtime chat or copilot serving with a P99 latency SLO A preemption mid-conversation drops the user; even seconds of unavailability blow tight SLOs and exhaust availability budgets.
Per-user pinned state services such as session-bound LoRAs or warm conversation caches Eviction destroys the pinned state and any rehydration takes minutes, far longer than spot's notice window.
Sole production replica with no warmed on-demand failover Spot is a discount tier, not a redundancy tier; a single preemption becomes an outage with no fallback.
Launch windows committed to 99.9 percent availability Even low preemption rates exhaust four-nines budgets quickly, so reserved or on-demand capacity is required during the launch.
Quick scan · 2–4 min
Everything important, quickly.
4 min: spot economics on H100 + the two conditions for spot-safety + why each wrong option fails + mixed-fleet architectural patterns + batch API monetisation.
Real-world examples
Real products, models, and research that use this idea.
1AWS EC2 p5 (H100) spot pricing sits around 50 to 70 percent below on-demand, used widely for overnight training and batch eval at large labs in 2026.
2Lambda Labs and CoreWeave offer interruptible H100 capacity primarily for fine-tuning and dataset labelling workloads with built-in checkpointing.
3Anthropic and OpenAI batch APIs run on heavily discounted internal spot tiers, exposing a 24-hour SLA that hides eviction recovery from callers.
4DeepSeek and Mistral training runs on spot H100 clusters routinely, using frequent checkpoint-and-restart loops to absorb preemptions.
5Production serving fleets at scale (chat, copilots) stay on on-demand or provisioned-throughput; spot only ever appears as a clearly separate batch lane.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QCould you serve customer traffic on spot at all, given a robust failover layer?
Sign in or go Pro to see the approach for this follow-up.
QHow does the batch API product from major LLM vendors relate to spot capacity?
Sign in or go Pro to see the approach for this follow-up.
QWhy is checkpoint frequency a knob worth tuning on spot training?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating spot as a generic cost lever and pointing it at customer-facing traffic. The discount is real, but the preemption probability eats the SLO long before the savings show up on the bill.