Bpe

Two providers both advertise $0.50 per million output tokens, why is that comparison still dishonest?

Bytes, characters, tokens: which one gets billed and how do they relate?

BPE learned 50,000 merge rules during training. What is a merge rule, and where does it live at inference?

Your teammate mentions BPE at a code review. What are they talking about?

You are fine-tuning on Python code only. Should you reuse the base tokenizer, train a new one, or extend the vocab?

BPE and WordPiece both merge subwords. What is the one thing they disagree on?

BPE grows the vocabulary by adding merges. Unigram does the opposite. Explain the difference.

How does WordPiece decide which pair to merge, compared to BPE?

How would you train a custom tokenizer for a genomics LLM on DNA sequences?

Explain the Unigram LM tokenization algorithm and how its training differs fundamentally from BPE. What does stochastic tokenization enable?

What primarily causes the 'token tax' for low-resource languages in BPE tokenizers?

Explain why word-level and character-level tokenization both fail for modern LLMs, and how subword tokenization resolves each failure mode.

Why do modern LLMs use subword tokenization instead of word-level or character-level approaches?

Which tokenizer can learn merges that span whitespace boundaries by design?

What is the primary purpose of regex-based pre-tokenization in cl100k_base, and what constraint does it impose on BPE?

Spot the error: 'BPE tokenizers can produce [UNK] when they encounter rare words'

Why do LLMs frequently fail at arithmetic involving multi-digit numbers? Trace the failure to tokenization.

Predict how cl100k_base tokenizes '1024', '2048', and '100000', and explain how inconsistent splitting causes LLM arithmetic failures.

Approximately what fraction of BPE vocabulary tokens does LiteToken identify as 'merge residues'?

What are BPE merge residues (LiteToken) and what do they imply for vocabulary efficiency?

Spot the tokenization boundary bug in this manual prompt-building code.

Predict whether tokenizer.encode('world') and tokenizer.encode(' world') produce the same token IDs in tiktoken.

What most inflates token count for European languages compared to English in a BPE tokenizer trained on English-dominant data?

Predict the token count difference between 'def fibonacci(n):' and 'def f(n):' in cl100k_base.

Why does a 4-space Python indentation block often tokenize to a single token in cl100k_base?