Evaluation

Offline eval scores 0.9 but users report frequent wrong answers: why the gap, and what do you do?

Compute recall@5 and MRR for this labeled retrieval result

Select every metric that belongs on a serious multi-agent eval dashboard

How do you watch answer quality in production when nothing is labeled?

Why gate every prompt or model change behind a held out eval in CI?

What's the right way to roll out a prompt change you believe is better?

Walk through the triage order when a customer support LoRA tanks MMLU by 4 points post-deploy

Why does using the same model family as synthetic data teacher AND eval judge inflate FT scores?

PremiumWhat is the root…

What is the diagnosis when a distilled model hits 92% on a public benchmark but 31% on an internal held-out set?

What cheap but predictive eval do you log during training?

PremiumWhat is it called…

A teammate asks what 'running evals' means in the LLM context. How would an engineer explain it without jargon?

How should you score a multi-step agent where final answer accuracy is 15% but most steps are correct?

Describe a concrete, production runnable mechanism to detect hallucinations in a RAG answer (claims unsupported by the retrieved chunks), as the answer is generated or just after.

Design a golden eval set for a production RAG system serving a legal research product. Explain how you build it, how big it should be, what each example contains, and what you measure with it.

What leakage modes in instruction tuning evaluation are easy to miss?

Which of these are real train/test leakage modes for an instruction tuning project?

Which of these belong on the ship checklist for a production fine-tuned model?

How do you design the post-FT eval suite for a customer support fine-tuned model?

Which buckets MUST a post-FT eval suite include for a customer support fine-tuned model?

PremiumHow do you detect…

Which signal is the most reliable indicator of catastrophic forgetting?

What is STRR and how does it complement fertility as a tokenizer fairness metric?

Match each RAGAS metric to what it specifically measures about a RAG pipeline.