Multiple Choice

Evaluating a code generation model on real-world tasks beyond HumanEval: which metrics cover correctness, efficiency, and style?

Automated eval vs human eval: which is better for catching subtle factual errors in medical text?

HELM from Stanford claims to be a 'holistic' evaluation. What makes it different from running MMLU alone?

Why does Chatbot Arena (LMSYS) give a different picture of model quality than static benchmarks like MMLU?

AlpacaEval 2 reports a 'win rate' for each model. Against what baseline and how is the winner decided?

Running LLM-as-judge on your full eval set costs $200 per run. The team runs evals 10 times a day during development. How do you cut cost without losing signal?

Your eval shows Model B scores 82.1% vs Model A at 80.5% on 200 test cases. Your manager wants to ship Model B immediately. What question should you ask first?

Why do teams still pay for human evaluation when LLM-as-judge exists?

In classical ML the data splits into train/val/test. For LLM evaluation, which split matters most and why?

A single-turn eval shows your chatbot scores 4.2/5. But users complain it 'forgets what they said.' Which eval dimension is your single-turn setup missing?

You tell the model 'respond in exactly 3 bullet points' and it gives you 5. Which benchmark specifically targets this class of failure?

A colleague claims their model is 'grounded' because it cites sources. Explain why citation presence alone does not prove factual grounding.

You need to find an MCP server for Slack. Where do you look in 2026?

Which MCP transports are in active use in 2026, and when do you pick each?

An MCP tools/call result comes back. Which content types could it contain?

Your colleague says MCP and function calling do the same thing. Where are they wrong?

You need to count tokens locally. When do you reach for tiktoken versus SentencePiece?

Your GPT-5.5 call bills 30 percent more tokens than the user prompt plus system instruction. Where are the invisible tokens?

You are fine-tuning Llama 3 8B on variable-length instruction data. Which padding strategy and side do you pick?

You need the exact token count for a Claude Sonnet 4.6 prompt before sending it. Which method actually works?

You are fine-tuning on Python code only. Should you reuse the base tokenizer, train a new one, or extend the vocab?

BPE and WordPiece both merge subwords. What is the one thing they disagree on?

Your indexing pipeline tokenizes 10M docs in 18 hours. What single change gives the biggest speedup?

How is a team of agents different from one agent with a large tool belt?

How does function calling enable the observe-plan-act agent loop?