Anyscale

What is RLHF, and why is it used after pretraining?

A teammate asks what 'running evals' means in the LLM context. How would an engineer explain it without jargon?

How do you evaluate a customer service chatbot that handles multi-turn conversations? What dimensions matter?

A consumer facing LLM product is about to launch. Describe the safety evaluation that should run first.

Automated eval vs human eval: which is better for catching subtle factual errors in medical text?

Identify the biggest risk when using LLM-as-judge as your only quality signal

You tell the model 'respond in exactly 3 bullet points' and it gives you 5. Which benchmark specifically targets this class of failure?

Why does restating the user question right before the answer position help with long contexts?

The PM asks how to tell if the chatbot got better after the last deploy. Show setting up an A/B test.

Writing the LLM-as-judge prompt for an eval pipeline: name three biases to design against and one mitigation each.