Tokenization

103 questions

How text becomes tokens. BPE, SentencePiece, WordPiece, vocabulary tradeoffs, special tokens, and why tokenizer choice matters at inference time.

Sort

Questions

What fraction of a 7B transformer lives in its embedding table at vocab 128k?
Multiple ChoiceMedium·Qual 4.0
Diagnose the reasoning error in 'a bigger tokenizer vocabulary always cuts our token bill'
Spot the ErrorMedium·Qual 4.0
Bytes, characters, tokens: which one gets billed and how do they relate?
FlashcardEasy·Qual 4.0
Defend pinning the tokenizer revision alongside the model weights when shipping a fine-tune
Short AnswerMedium·Qual 4.0
Pick the most accurate one-sentence summary of the tokenizer's role in fine-tuning.
Multiple ChoiceEasy·Qual 4.0
A tokenizer has a 'vocabulary'. What is it, and why does its size matter for the model?
FlashcardEasy·Qual 4.0
Your team debates 32K vs 128K vs 256K vocab. What is the core tradeoff they should frame?
FlashcardEasy·Qual 4.0
Your input is 9,000 tokens but max_length is 8,192. What are your truncation options?
FlashcardEasy·Qual 4.0
You open a HuggingFace model repo and see tokenizer_config.json. What is in it and what is NOT?
FlashcardEasy·Qual 4.0
Token 1234 means 'the' in one model and garbage in another. Why?
FlashcardEasy·Qual 4.0
An LLM never sees raw text. So what does it actually process?
FlashcardEasy·Qual 4.0
BOS, EOS, PAD: why do tokenizers reserve these ids instead of using normal text?
FlashcardEasy·Qual 4.0
Someone confuses 'sequence length' with 'context window'. How do you correct them?
FlashcardEasy·Qual 4.0
You change one word in the system prompt and prompt caching breaks. Why?
FlashcardEasy·Qual 4.0
Pre-tokenization splits text before BPE merges. Why is that boundary important?
FlashcardEasy·Qual 4.0
You pad a batch to equal length. How does the model know to ignore the filler tokens?
FlashcardEasy·Qual 4.0
Older tokenizers had an OOV problem. Why did byte-level BPE kill it?
FlashcardEasy·Qual 4.0
Before the tokenizer even starts splitting, normalization runs. What does it do and why?
FlashcardEasy·Qual 4.0
BPE learned 50,000 merge rules during training. What is a merge rule, and where does it live at inference?
FlashcardEasy·Qual 4.0
You hear 'fertility is 2.5 for Hindi on cl100k_base'. What does that number mean and is it good?
FlashcardEasy·Qual 4.0
A tokenizer is 'byte-level'. What does that buy you over a character-level one?
FlashcardEasy·Qual 4.0
Your teammate mentions BPE at a code review. What are they talking about?
FlashcardEasy·Qual 4.0
A chat template wraps every turn with BOS and EOS tokens. Why does the model need them?
FlashcardEasy·Qual 4.0
You want to add 5,000 Hindi medical terms to Llama 3 8B. Walk through the procedure and the risks.
Short AnswerMedium·Qual 4.0
Self-hosted Llama 3 chat returns fluent-looking nonsense. You check and the prompt was tokenized with cl100k_base. What went wrong?
Spot the ErrorMedium·Qual 4.0

Showing 1–25 of 103

← PrevNext →