LLM As Judge

Identify the judge bias risk when the same model judges its own multi-agent trajectories

Why does using the same model family as synthetic data teacher AND eval judge inflate FT scores?

LLM-as-judge scores drift up across 50k weekly evals while user complaints hold steady. Diagnose likely causes and design a calibration system preventing silent score inflation.

Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per product customization, and six month failure modes.

Pick the statement that best describes a scoring rubric in LLM evaluation.

Before running LLM-as-judge, someone insists on writing a rubric first. What is a rubric in this context?

When someone says 'eval prompt template,' what are they referring to and why does the wording matter so much?

Your LLM judge gives everything a 4 out of 5. Your colleague says the judge is not calibrated. What does calibration mean here?

You have no reference answers for your eval set. Does that mean you cannot evaluate the model?

Explain the difference between pointwise and pairwise evaluation in one breath.

ROUGE-L, BERTScore, and LLM-as-judge are on the table for a summarization eval. Which covers what, and which drops first if budget is tight?

Writing the LLM-as-judge prompt for an eval pipeline: name three biases to design against and one mitigation each.

Evaluating a code generation model on real world tasks beyond HumanEval: which metrics cover correctness, efficiency, and style?

How do you evaluate a customer service chatbot that handles multi-turn conversations? What dimensions matter?

PremiumRunning LLM-as-judge on your…

A production model has no ground truth labels on live traffic. Design a monitoring plan that catches degradation before users complain.

Define LLM-as-judge and explain the one problem it was invented to solve.

Non-obvious modes of train/test leakage in instruction tuning evaluation

Which of these are real train/test leakage modes for an instruction tuning project?

Where does quality leak in a self-instruct synthetic data pipeline?

Design the post-FT eval suite for a customer support fine-tune

When is LLM-as-judge an appropriate evaluation method for prompt outputs and what are its known biases that you need to mitigate?

PremiumHow do you design…

Explain why string match accuracy is systematically misleading for LLM evaluation

Why string match accuracy fails as an LLM evaluation metric