Scale AI

Design a diagnostic eval for an agentic workflow with 62% task success across 5 tools and 10+ steps that reveals WHY tasks fail, not just whether they pass.

LLM-as-judge scores drift up across 50k weekly evals while user complaints hold steady. Diagnose likely causes and design a calibration system preventing silent score inflation.

Design a multi-dimensional eval dashboard for a production LLM product. What dimensions, views, and alerts does it need, and what is the one chart that the VP of Engineering should check every Monday?

Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per-product customization, and six-month failure modes.

A colleague presents this claim: 'Our chatbot improved because BLEU score went from 0.32 to 0.41 on our eval set of open-ended customer questions.' Spot the flaw in this reasoning.

A model aces a benchmark with suspiciously high scores. The first hypothesis is contamination. Explain what that means to a non-technical PM.

Your LLM judge gives everything a 4 out of 5. Your colleague says the judge is not calibrated. What does calibration mean here?

BLEU score comes up in a meeting. What does it measure and when does it still make sense?

Every model release cites benchmark numbers. Define what a benchmark actually is and name two things it cannot tell you.

You have no reference answers for your eval set. Does that mean you cannot evaluate the model?