Zenaique
Topics
Practice
Browse
Reference
Challenges
Pricing
Search…
⌘K
Topics
/
Tokenization
/
Pre Tokenization Regex
Pre Tokenization Regex
Subtopic
4 questions
Questions tagged with Pre Tokenization Regex — part of Tokenization.
Format
Difficulty
Role
Experience
Companies
Sort
Newest
Quality
Difficulty ↑
Difficulty ↓
Questions
Why does SentencePiece prepend a special space symbol (▁) to tokens, and what would break without it?
Short Answer
Hard
·
Qual 4.0
Google
Which tokenizer can learn merges that span whitespace boundaries by design?
Multiple Choice
Medium
·
Qual 4.0
Google
What happens to emoji ZWJ sequences under cl100k_base vs. o200k_base pre-tokenization?
Short Answer
Hard
·
Qual 4.0
OpenAI
What is the primary purpose of regex-based pre-tokenization in cl100k_base, and what constraint does it impose on BPE?
Multiple Choice
Medium
·
Qual 4.0