Vocabulary

Pick the most accurate one-sentence summary of the tokenizer's role in fine-tuning.

How does softmax turn an attention score into an attention weight?

Decode the acronym MHA in the transformer attention context.

Identify the context vector inside scaled dot-product attention and what produces it.

A tokenizer has a 'vocabulary'. What is it, and why does its size matter for the model?

Your teammate mentions BPE at a code review. What are they talking about?

Flashcard: what does zero-shot prompting mean and what makes it work at all?

Flashcard: what is a vector database, and why does RAG need one?

Flashcard: what is similarity search in RAG, and how does it differ from keyword search?

Flashcard: what does 'grounding' mean in the context of RAG, and why is it the whole point?

Flashcard: what is an embedding in a RAG pipeline, and what role does it play?

Flashcard: what does chunking mean in RAG, and why do we chunk at all?

Your team is comparing a vanilla LLM chatbot against a RAG chatbot. What is the single most important difference between them at query time?

Flashcard: what role does the LLM's context window play in a RAG pipeline?

Select all components that are required to build a minimal end to end RAG system (the absolute basics, not nice to haves like rerankers or eval harnesses).

How does vocabulary size affect embedding table memory footprint at bfloat16 precision, and what are the tradeoffs of very small vs. very large vocabularies?

Which part of a transformer model's parameter count scales directly and linearly with vocabulary size?

What concretely breaks when you try to load a pretrained model checkpoint with a different tokenizer vocabulary?

Why is it not possible to simply swap a pretrained LLM's tokenizer for one with a larger vocabulary?

How does weighting BPE merges by document frequency instead of raw token frequency change the resulting vocabulary?