Spot the Error

Spot the conceptual error: 'We're using PQ instead of HNSW for our vector index because PQ is faster.'

Spot the error in this candidate's description of IVF's two main parameters.

Spot the error: 'Pre-norm and post-norm are interchangeable, just a stylistic choice'

Spot the error: 'The FFN mixes information across tokens'

TPOT is high, so the team plans to upgrade A100 to a faster compute GPU, critique

Diagnose the reasoning error in 'a bigger tokenizer vocabulary always cuts our token bill'

'Prompt caching cuts our bill by 90 percent across the board', what's overclaimed?

Find the wrong move in 'decode is slow, so let's switch to a smaller-FLOP model'

A teammate randomized system prompts per user and expects the prompt cache to still help, what's wrong?

Spot the error: 'we autoscale our LLM serving pods on CPU utilization.'

Spot the bug: packed-sequence SFT where the model learns to copy across unrelated examples

Spot the bug: chat-template code that drops the EOS from the labels.

DataCollator pads to max_length and sets attention_mask = ones: find what breaks

Predict what softmax produces when every key in a row is masked out.

A batch packs a causal LM with right-padded sequences and applies only the causal mask. Spot the mistake.

Spot the masking bug in this packed-sequence training setup

Spot the GQA configuration error: num_attention_heads=32, num_key_value_heads=7.

Spot the flaw: a claim that attention by itself notices position.

Critique this claim about CLIP using cross-attention between towers

Find the bug: a Llama deployment serves traffic without prepending the BOS token.

A colleague presents this claim: 'Our chatbot improved because BLEU score went from 0.32 to 0.41 on our eval set of open-ended customer questions.' Spot the flaw in this reasoning.

Spot the flaw: 'We fine-tuned on 10K examples and eval on a random 500 from the same dataset. Accuracy is 94%, so the model is ready for production.'

Self-hosted Llama 3 chat returns fluent-looking nonsense. You check and the prompt was tokenized with cl100k_base. What went wrong?

A hand-built chat prompt performs worse than apply_chat_template. Spot the tokenization issues.

After editing the Jinja chat template, your fine-tuned Llama 3 prints raw special-token text. What broke?