Don't say this

Tokenization
An LLM never sees raw text. So what does it process?
FlashcardEasyCoinbaseMongodb
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Most common mistake
Treating tokens as words or as characters. Tokens are sub-word units defined by the tokenizer, and the same string can produce different token counts under different tokenizers.
Sign in to see all red flags and common mistakes.
Open full question
Character, word, and subword tokenization each have a dominant failure mode. Match them.
Match PairsEasyCursorTurbopuffer
Left and right
Character level tokenization
Word level tokenization
Subword tokenization (BPE, WordPiece, Unigram LM)
Byte level subword tokenization (byte level BPE)
Closed vocabulary; any unseen word becomes [UNK] and information is lost
No native failure mode at this resolution; this is the Pareto compromise modern LLMs use
Latin script bias: non-Latin scripts often need many bytes per character, inflating token counts and costs
Sequence length explosion that blows up attention cost and shrinks effective context
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Most common mistake
Claiming subword tokenization has no failure mode at all. Byte-level subword has a real, measurable Latin-script bias because non-Latin code points often encode to multiple bytes, inflating token counts for those languages.
Sign in to see all red flags and common mistakes.
Open full question
Why do modern LLMs use subword tokenization instead of word level or character level approaches?
Multiple ChoiceEasyIntuitMeta
Options
Word level tokenizers fail on new words not seen during training, while character level tokenizers produce sequences far too long for attention to process efficiently.
Subword tokenizers always produce shorter sequences than word level tokenizers for the same input text.
Character level tokenizers require a larger vocabulary than subword tokenizers, making them memory inefficient.
Word level tokenizers use too much memory because each word needs its own embedding vector.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Most common mistake
Claiming subword always yields shorter sequences than word-level, when in fact word-level is shortest and subword trades a little length to gain open-vocabulary coverage.
Sign in to see all red flags and common mistakes.
Open full question
PremiumExplain why word level…
Short AnswerMediumGoogleMeta
Unlock with Pro to study this question. See Pro pricing.
Open full question
How many tokens does "hello world" use with a GPT tokenizer?
Predict OutputEasyFiddler AiPinterest
Using a standard GPT style BPE tokenizer on the string:

'hello world'

Approximate token count?
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Most common mistake
Counting characters or assuming the leading space splits off as its own token. Modern GPT-style BPE bundles a leading space with the following word.
Sign in to see all red flags and common mistakes.
Open full question
PremiumSpot the error in…
Spot the ErrorMediumAirbnbBaseten
Unlock with Pro to study this question. See Pro pricing.
Open full question
Which Unicode normalization does SentencePiece apply by default, and how does it differ from NFC?
Multiple ChoiceMediumDecagonSambanova
Options
SentencePiece applies NFKC normalization, which additionally decomposes compatibility characters (half width katakana, ligatures, circled numbers) unlike NFC which only handles canonical equivalences.
SentencePiece applies NFC normalization, which is identical to NFKC for all practical purposes since both produce the same byte sequences for standard text.
SentencePiece applies no Unicode normalization and passes raw bytes directly to its segmentation model.
SentencePiece applies NFD normalization, which decomposes composed characters into their base + combining mark form before tokenization.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Most common mistake
Assuming NFC and NFKC are interchangeable because they agree on plain ASCII: they diverge on any ligature, half-width, or circled character, which is exactly where multilingual bugs hide.
Sign in to see all red flags and common mistakes.
Open full question
PremiumSpot the failure mode:…
Spot the ErrorHardSalesforceVellum
Unlock with Pro to study this question. See Pro pricing.
Open full question
Before the tokenizer even starts splitting, normalization runs. What does it do and why?
FlashcardEasyAdaWorkday
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Most common mistake
Assuming all tokenizers normalize the same way. tiktoken does almost nothing; Hugging Face tokenizers vary widely. Mismatched normalization between training and inference produces silent drift in token sequences.
Sign in to see all red flags and common mistakes.
Open full question
PremiumWhat is the primary…
Multiple ChoiceMediumBanana DevElastic
Unlock with Pro to study this question. See Pro pricing.
Open full question

Showing 1–10 of 2404

← PrevNext →

Tokenization