Evaluation

Offline eval scores 0.9 but users report frequent wrong answers: why the gap, and what do you do?

Compute recall@5 and MRR for this labeled retrieval result

Select every metric that belongs on a serious multi-agent eval dashboard

How do you watch answer quality in production when nothing is labeled?

Why gate every prompt or model change behind a held out eval in CI?

What's the right way to roll out a prompt change you believe is better?

Walk through the triage order when a customer support LoRA tanks MMLU by 4 points post-deploy

Why does using the same model family as synthetic data teacher AND eval judge inflate FT scores?

PremiumWhen eval prompts appear…

A distilled model hits 92% on a public benchmark but 31% on an internal hold out: diagnose.

Picking a cheap but predictive eval to track during training: what do you actually log?

PremiumTrain loss drops while…

A teammate asks what 'running evals' means in the LLM context. How would an engineer explain it without jargon?

How should you score a multi-step agent where final answer accuracy is 15% but most steps are correct?

Describe a concrete, production runnable mechanism to detect hallucinations in a RAG answer (claims unsupported by the retrieved chunks), as the answer is generated or just after.

Design a golden eval set for a production RAG system serving a legal research product. Explain how you build it, how big it should be, what each example contains, and what you measure with it.

Non-obvious modes of train/test leakage in instruction tuning evaluation

Which of these are real train/test leakage modes for an instruction tuning project?

Which of these belong on the ship checklist for a production fine-tune?

Design the post-FT eval suite for a customer support fine-tune

Which buckets MUST a post-FT eval suite include for a customer support fine-tune?

PremiumHow do you detect…

Which signal is the most reliable indicator of catastrophic forgetting?

What is STRR and how does it complement fertility as a tokenizer fairness metric?

Match each RAGAS metric to what it specifically measures about a RAG pipeline.