Cl100k

What happens to emoji ZWJ sequences under cl100k_base vs. o200k_base pre-tokenization?

What is the primary purpose of regex-based pre-tokenization in cl100k_base, and what constraint does it impose on BPE?

Predict how cl100k_base tokenizes '1024', '2048', and '100000', and explain how inconsistent splitting causes LLM arithmetic failures.

Predict the token count difference between 'def fibonacci(n):' and 'def f(n):' in cl100k_base.

Why does a 4-space Python indentation block often tokenize to a single token in cl100k_base?

What does the '200k' in o200k_base signify, and what was improved compared to cl100k_base?