Flashcard

MQA, spell out the acronym and name exactly what gets shared across heads

Decode the acronym MHA in the transformer attention context.

Across which axes is the KV cache stored, per layer, per head, per batch?

Compare hidden_size, d_model, and embedding_dim, three names for the same thing or three different things?

GQA, decode the acronym and describe its KV sharing pattern

What does the dot product q . k actually measure inside an attention layer?

Define a single attention head, what does one head actually own and produce?

ALiBi, decode the acronym and pinpoint where in the pipeline it injects position

Before running LLM-as-judge, someone insists on writing a rubric first. What is a rubric in this context?

ROUGE keeps showing up in summarization evals. How does it work and where does it break?

A model card reports perplexity of 3.2. Explain what that number means in plain English.

When someone says 'eval prompt template,' what are they referring to and why does the wording matter so much?

A teammate asks what 'running evals' means in the LLM context. How would an engineer explain it without jargon?

A model aces a benchmark with suspiciously high scores. The first hypothesis is contamination. Explain what that means to a non-technical PM.

Your LLM judge gives everything a 4 out of 5. Your colleague says the judge is not calibrated. What does calibration mean here?

BLEU score comes up in a meeting. What does it measure and when does it still make sense?

Every model release cites benchmark numbers. Define what a benchmark actually is and name two things it cannot tell you.

Explain the difference between pointwise and pairwise evaluation in one breath.

TruthfulQA is designed to catch a specific model failure. Describe what it tests and why larger models sometimes do worse.

MT-Bench is used alongside Chatbot Arena. Describe what MT-Bench tests that single-turn benchmarks miss.

MMLU appears on every model leaderboard. Describe what it tests and its biggest blind spot.

HumanEval is the go-to code generation benchmark. Describe the task it gives the model and how it decides if the answer is correct.

Chatbot Arena ranks models by Elo rating. Explain the Elo system to someone who has never played competitive chess.

You are building a golden dataset with three annotators. Someone warns you about inter-annotator agreement. What is it and what goes wrong if you skip measuring it?

Define LLM-as-judge and explain the one problem it was invented to solve.