Evaluation

Walk through the triage order when a customer-support LoRA tanks MMLU by 4 points post-deploy

Why does using the same model family as synthetic-data teacher AND eval judge inflate FT scores?

When eval prompts appear verbatim inside an SFT file after a synthetic-data run, what's the root cause?

A distilled model hits 92% on a public benchmark but 31% on an internal hold-out: diagnose.

Picking a cheap but predictive eval to track during training: what do you actually log?

Train loss drops while eval loss climbs at step 1200: name it and the standard fixes

A teammate asks what 'running evals' means in the LLM context. How would an engineer explain it without jargon?

How should you score a multi-step agent where final answer accuracy is 15% but most steps are correct?

Describe a concrete, production runnable mechanism to detect hallucinations in a RAG answer (claims unsupported by the retrieved chunks), as the answer is generated or just after.

Design a golden eval set for a production RAG system serving a legal research product. Explain how you build it, how big it should be, what each example contains, and what you measure with it.

Non-obvious modes of train/test leakage in instruction-tuning evaluation

Which of these are real train/test leakage modes for an instruction-tuning project?

Which of these belong on the ship-checklist for a production fine-tune?

Design the post-FT eval suite for a customer-support fine-tune

Which buckets MUST a post-FT eval suite include for a customer-support fine-tune?

How do you detect catastrophic forgetting after a domain fine-tune?

Which signal is the most reliable indicator of catastrophic forgetting?

What is STRR and how does it complement fertility as a tokenizer fairness metric?

Match each RAGAS metric to what it specifically measures about a RAG pipeline.

Your RAG system's faithfulness metric just dropped from 0.91 to 0.74 while context recall stayed flat at 0.85. What's the most likely pipeline layer to investigate first?

Complete the claim: the two-axis decomposition of RAG eval that diagnoses which layer failed.

What criteria would you use to evaluate the quality of an MCP server's tool definitions before integration?

How can an agent confidently produce a wrong answer without raising any runtime exception?