Alert on (cost > per-user p95) AND (cost > N times yesterday), with an absolute floor and a per-user cooldown. Calendar-aware, dedupe by user.
Picture monitoring electricity bills for a thousand apartments. You want to spot the one apartment that suddenly tripled its usage and ignore the dozens that legitimately turned on the heating on the first cold day of winter. Two checks help. First, compare each apartment to its own normal: a normally-quiet flat using twice as much is suspicious in a way that a normally-busy flat using the same amount is not. Second, compare today to yesterday: if everyone in the building went up together, it is probably the weather, not a leaky pipe. You only call the plumber when both signals fire at once. The cost per user alert works the same way.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Cost alerts are deceptively simple to ask for and surprisingly subtle to design. 'Just page me if a user spends too much' assumes a stable definition of 'too much'. In LLM products with mixed user populations (free trials, paid plans, enterprise contracts, internal bots, API consumers), no single number works.
The design pattern below is what teams converge on after a few false starts. It is also a useful general lesson in alert design: layer multiple signals, each addressing a specific failure mode, rather than try to make one threshold work for everything.
Per-user cost aggregation is the foundation
Everything downstream depends on a clean per-user cost time-series. Each LLM span carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, plus your computed USD cost using current pricing. Tag each span with a stable opaque user_id (NOT email; email is high-cardinality PII).
Roll the spans up into per user per day cost buckets. Most observability backends (Langfuse, Phoenix, Datadog) can do this natively; if not, a scheduled job that materializes the rollup table is straightforward.
The single biggest source of bugs in this whole design is missing or inconsistent user attribution. If half your spans have user_id and half do not, your per-user analytics are wrong in invisible ways. Set the user_id in a shared SDK wrapper or middleware once, not at every call site, and audit the coverage.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse 3.x supports per-user cost rollups via the user attribution attribute and offers cost-alerting integrations.
- Helicone exposes per-user spend dashboards with configurable rate of change alerts.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you extend this to detect slow-burn abuse over weeks instead of single-day spikes?
Talk about rolling 7-day growth rate, cumulative spend over a billing window, and dashboards for trend detection.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Using a single global cost threshold. Power users blow past it daily; quiet users never trigger it even when their usage actually doubled overnight.
60 second bullets to scan on the way to the call.
Explain why a single global cost threshold fails in both directions
Describe the role of a per-user p95 baseline
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.