A mature LLM cost dashboard distributes spend across requests, tenants, routes, and time, plus tracks cache-hit rate; hardware telemetry like GPU temperature belongs on an infra dashboard, not the cost view.
Picture the monthly electricity bill for an office building. You do not want one giant number at the end of the month. You want to know which floors use the most power, which appliances spike during the day, whether the new energy-saving panels are actually working, and whether the running total this month is tracking the budget or running ahead. You also want to spot the one tenant whose printer is somehow drawing as much as the rest of the floor. None of that needs the temperature of the power plant burning the coal a thousand miles away. The cost dashboard answers the questions an accountant or a product manager would ask. The hardware dashboard answers the questions an engineer with a wrench would ask. They are different tools for different audiences.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A cost dashboard for an LLM-backed product is read by three different audiences with three different questions. Engineering wants to know where the long-tail spend lives and whether their optimization work is paying off. Product wants to know which features cost what so it can make 'should we ship this' decisions on a real budget. Finance wants to know whether the monthly run-rate is tracking forecast or quietly drifting toward an invoice surprise.
The trap most teams fall into is treating the cost dashboard as a place to dump every metric the LLM platform emits. The dashboard ends up with thirty tiles, half of them infra telemetry, and the actual cost signal gets lost. A disciplined cost dashboard is short, opinionated, and groups its tiles by audience.
The sections below cover what belongs on each panel, why GPU and hardware metrics are deliberately excluded, and the two alerts that pair with the dashboard so attention is routed in real time instead of waiting for a weekly review.
The four panels and what each one shows
Panel one is the distribution panel. Per-request cost at p50, p95, and p99 plus token counts at the same percentiles. Average alone is a misleading summary because one runaway request can dominate the bill while leaving the mean nearly unchanged. The percentile triple makes the outlier visible.
Panel two is the attribution panel. Cost per tenant per day with a top-N table is the centerpiece, because a single abusive integration is the most common cause of a surprise invoice. Cost per intent or per route is the other half of this panel; it answers 'which product features cost what', which is the framing product managers can act on.
Panel three is the optimization-payoff panel. Cache-hit rate on cached input tokens, router-decision distribution (how often each model is chosen), and cost per cohort (free vs paid, by region) are the metrics that justify continued investment in cost work. Without them the team that shipped the caching project last month has no way to defend the investment when budget comes up for renewal.
Panel four is the forecast panel. Cumulative month to date spend overlaid with the monthly budget line, plus a projected end of month spend with confidence bands. This is the panel finance opens first and engineering opens last; both need to see the same number.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse and Helicone both ship per-tenant and per-model cost views out of the box that satisfy the attribution rule without custom dashboarding work.
- LiteLLM Proxy exposes budgets per virtual key, so per-tenant burn-rate alerts can fire automatically when an integration runs past its allocation.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you implement cost attribution per tenant without per-tenant API keys at the provider?
Run a gateway like LiteLLM or Portkey that tags each request with a tenant id, computes cost from the response's usage block, and writes it to a per-tenant counter the dashboard reads.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Mixing infra telemetry (GPU temp, memory, utilization) onto the cost dashboard. They belong on a separate serving dashboard so the cost view stays readable to finance and product.
60 second bullets to scan on the way to the call.
Per-request cost at p50, p95, p99
Per-tenant per-day burn rate with top-N panel
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.