Fresher / New Grad

HumanEval is the go-to code generation benchmark. Describe the task it gives the model and how it decides if the answer is correct.

HELM from Stanford claims to be a 'holistic' evaluation. What makes it different from running MMLU alone?

Chatbot Arena ranks models by Elo rating. Explain the Elo system to someone who has never played competitive chess.

Why does Chatbot Arena (LMSYS) give a different picture of model quality than static benchmarks like MMLU?

AlpacaEval 2 reports a 'win rate' for each model. Against what baseline and how is the winner decided?

You are building a golden dataset with three annotators. Someone warns you about inter-annotator agreement. What is it and what goes wrong if you skip measuring it?

Why do teams still pay for human evaluation when LLM-as-judge exists?

Define LLM-as-judge and explain the one problem it was invented to solve.

You hear about 'eval harnesses' like Promptfoo and DeepEval. Explain what an eval harness does that a Jupyter notebook cannot.

In classical ML the data splits into train/val/test. For LLM evaluation, which split matters most and why?

Before shipping a prompt change, the lead says 'run it against the golden set.' What is a golden dataset and why does every team need one?

Where do you configure MCP servers in Zed, and where do their tools appear?

You want to build an MCP server in TypeScript. What package do you use and how does it handle schemas?

Walk me through what tools/list returns and when the host calls it.

Your MCP tool just failed. How does the server tell the model what went wrong?

You need to find an MCP server for Slack. Where do you look in 2026?

You want to build an MCP server in Python. What do you install and what does the API look like?

In MCP, how is a notification different from a request?

Which MCP transports are in active use in 2026, and when do you pick each?

In MCP, what exactly is a 'tool' and who decides when to call it?

Your teammate says they built an 'MCP server'. What did they actually build?

MCP has tools, resources, and prompts. What makes a resource different from a tool?

Why does MCP have a 'prompt' primitive separate from tools and resources?

Your MCP server builds but the host can't see its tools. What's the first debugging move?

How is an MCP host different from an MCP client?