AI Product Manager

For a machine translation eval, a senior engineer says BLEU is fine as the primary metric. When are they right, and when should you push back?

ROUGE-L, BERTScore, and LLM-as-judge are on the table for a summarization eval. Which covers what, and which drops first if budget is tight?

A RAG pipeline just shipped for internal docs. How would the team know if retrieval is actually helping?

How do you evaluate a customer service chatbot that handles multi-turn conversations? What dimensions matter?

Automated eval vs human eval: which is better for catching subtle factual errors in medical text?

An 18-month-old golden eval set, departed annotators, three model updates, shifted production traffic. Design the lifecycle management process.

MMLU appears on every model leaderboard. Describe what it tests and its biggest blind spot.

Chatbot Arena ranks models by Elo rating. Explain the Elo system to someone who has never played competitive chess.

Why does Chatbot Arena (LMSYS) give a different picture of model quality than static benchmarks like MMLU?

AlpacaEval 2 reports a 'win rate' for each model. Against what baseline and how is the winner decided?

A consumer-facing LLM product is about to launch. Describe the safety evaluation that should run first.

Running LLM-as-judge on your full eval set costs $200 per run. The team runs evals 10 times a day during development. How do you cut cost without losing signal?

The PM asks how to tell if the chatbot got better after the last deploy. Walk through setting up an A/B test.

Your eval shows Model B scores 82.1% vs Model A at 80.5% on 200 test cases. Your manager wants to ship Model B immediately. What question should you ask first?

Building a golden dataset for RAG eval from scratch: walk through the steps and name the mistake that wastes the most labeling budget.

You are building a golden dataset with three annotators. Someone warns you about inter-annotator agreement. What is it and what goes wrong if you skip measuring it?

Why do teams still pay for human evaluation when LLM-as-judge exists?

In classical ML the data splits into train/val/test. For LLM evaluation, which split matters most and why?

Before shipping a prompt change, the lead says 'run it against the golden set.' What is a golden dataset and why does every team need one?

Someone asks you what MCP is at a meetup. How do you explain it?

A 40-team org already has OpenAPI specs and three LLM providers. Defend layering MCP on top instead of picking one integration approach.

Your colleague says MCP and function calling do the same thing. Where are they wrong?

You are building a 7B model for 25 languages including Hindi, Arabic, and Swahili. Design a fair tokenizer and name what it costs.

$50K compute budget, 50B tokens of US legal text, 1.5B parameter model. Defend your tokenizer design.

How should you score a multi-step agent where final answer accuracy is 15% but most steps are correct?