Design a multi-dimensional eval dashboard for a production LLM product. What dimensions, views, and alerts does it need, and what is the one chart that the VP of Engineering should check every Monday?
Design a multi-dimensional eval dashboard for a production LLM product. What dimensions, views, and alerts does it need, and what is the one chart that the VP of Engineering should check every Monday?
Five dimensions (quality, safety, cost, latency, user signals), four views (trend, slice, run comparison, alerts), composite quality trendline with deployment markers as VP Monday chart.
Imagine the dashboard on the wall of a hospital emergency room. One screen shows vital signs (quality), another shows infection incidents (safety), another shows supply costs (cost), another shows wait times (latency), and a feedback board shows patient satisfaction (user signals). The hospital director checks one chart every morning: a single line showing the overall health score with markers for when new staff started or new equipment arrived. If the line dips after a change, they know where to look. The most dangerous situation is when the overall score looks fine but one department is struggling, hidden by improvements elsewhere.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A production LLM eval dashboard is the team's primary interface for understanding whether the product is healthy. Unlike traditional software monitoring, which tracks uptime and error rates, LLM monitoring must track quality (a continuous, subjective metric), safety (a hard constraint), cost (which scales with usage in ways traditional software does not), latency (with LLM-specific metrics like time to first token), and user signals (the ground-truth check that model-based metrics cannot replace).
The design challenge is not displaying data. It is designing views and alerts that surface the right information to the right audience at the right time. The VP of Engineering needs one chart. The on-call engineer needs slice-level breakdowns. The product manager needs run comparisons for deployment decisions. The dashboard must serve all three without overwhelming any of them.
This walkthrough covers the five dimensions, four views, four alert types, and the VP Monday chart, with emphasis on the most dangerous failure mode in production: slice-level regression hidden by a stable aggregate.
The five measurement dimensions
Each dimension captures a different failure mode, which is why all five are necessary.
Quality is the model-based assessment of output correctness and usefulness. Metrics include LLM-as-judge scores (overall quality, faithfulness, relevance), task completion rate (did the model finish the task?), and format compliance (did the output follow the expected structure?). Quality is the dimension most teams instrument first, but it is also the most fragile because it depends on the judge's calibration.
Safety tracks hard constraints that must never be violated. Metrics include refusal rate (broken into true positives, where the model correctly refused, and false positives, where the model refused a safe query), toxicity incidents, PII leakage events, and policy violation count. Safety metrics use absolute thresholds, not statistical deviation: any PII leakage is critical regardless of the historical rate.
Cost tracks the financial dimension that traditional software monitoring ignores. Metrics include input tokens per query, output tokens per query (separated because they have different pricing), judge cost per eval run, total daily inference spend, and cost per successful task completion. Cost per successful completion is the most useful unit because it normalizes for retry rates and failed attempts.
Latency uses LLM-specific timing points. Time to first token (TTFT) measures how long the user waits before seeing any response, which drives perceived responsiveness. Total response time measures the full generation. Both are tracked at P50, P95, and P99, broken down by model tier and region. TTFT is the metric users feel most acutely.
User signals provide the ground-truth check that model-based metrics cannot replace. Thumbs-up and thumbs-down rates measure user satisfaction. Escalation rate (user asked for a human) measures confidence in the model. Repeat-query rate (user rephrased the same question) suggests the first answer failed. Session abandonment rate measures whether users gave up entirely. When user signals diverge from model-based quality scores, the model scores are wrong.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Arize Phoenix provides multi-dimensional LLM monitoring dashboards with per-trace quality, latency, and cost breakdowns, used by production teams to detect regressions across query categories.
- LangSmith offers run comparison views with per-metric confidence intervals, enabling teams to evaluate deployment candidates against last known good baselines.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you compute the composite quality score for the VP chart without making it gameable by teams optimizing one component at the expense of others?
Weight the components inversely to their current performance: the weakest dimension gets the highest weight, preventing teams from boosting an already-strong metric while ignoring a weak one. Alternatively, use a floor-based composite where the score is the minimum of all components, not the average. A min-based composite forces every dimension above threshold before the composite improves.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Monitoring only aggregate quality scores without slice-level breakdowns. A 15% regression in one query category can be hidden by improvements in another, making the aggregate look stable while a segment of users experiences real degradation.
60 second bullets to scan on the way to the call.
Name at least four dimensions: quality, safety, cost, latency, user signals
Describe the time-series trend view with deployment annotations
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.