Zenaique
Topics
Practice
Browse
Reference
Challenges
Pricing
Search…
⌘K
Topics
/
Tokenization
/
Regex
Regex
Subtopic
3 questions
Questions tagged with Regex — part of Tokenization.
Format
Difficulty
Role
Experience
Companies
Sort
Newest
Quality
Difficulty ↑
Difficulty ↓
Questions
Pre-tokenization splits text before BPE merges. Why is that boundary important?
Flashcard
Easy
·
Qual 4.0
What happens to emoji ZWJ sequences under cl100k_base vs. o200k_base pre-tokenization?
Short Answer
Hard
·
Qual 4.0
OpenAI
What is the primary purpose of regex-based pre-tokenization in cl100k_base, and what constraint does it impose on BPE?
Multiple Choice
Medium
·
Qual 4.0