Fresher / New Grad

An LLM never sees raw text. So what does it process?

Before the tokenizer even starts splitting, normalization runs. What does it do and why?

Pre-tokenization splits text before BPE merges. Why is that boundary important?

A tokenizer has a vocabulary. What is it, and why does its size matter for the model?

Token 1234 means "the" in one model and garbage in another. Why?

Your team debates 32K vs 128K vs 256K vocab. What is the core tradeoff they should frame?

You open a HuggingFace model repo and see tokenizer_config.json. What is in it and what is NOT?

BOS, EOS, PAD: why do tokenizers reserve these ids instead of using normal text?

What are BOS, EOS, PAD, and role marker special tokens, and what happens when each is missing or duplicated?

You pad a batch to equal length. How does the model know to ignore the filler tokens?

Which of bytes, characters, or tokens does an LLM API charge you for, and how do the three relate?

Your teammate mentions BPE at a code review. What are they talking about?

Why use subword tokens, and what is wrong with word level or character level?

BPE learned 50,000 merge rules during training. What is a merge rule, and where does it live at inference?

BPE grows the vocabulary by adding merges. Unigram does the opposite. Explain the difference.

A tokenizer is byte level. What does that buy you over a character level one?

How does byte level BPE guarantee no unknown tokens, and what is the cost of that guarantee?

Character, word, and subword tokenization each have a dominant failure mode. Match them.

Why do modern LLMs use subword tokenization instead of word level or character level approaches?

BPE and WordPiece both merge subwords. What is the one thing they disagree on?

PremiumExplain why word level…

How many tokens does "hello world" use with a GPT tokenizer?

PremiumSpot the error in…

BPE finds the globally optimal vocabulary by iteratively merging the most frequent pair.

PremiumPut the BPE training…