Zenaique
Topics
Practice
Browse
Reference
Challenges
Pricing
Search…
⌘K
Topics
/
Tokenization
/
Pre Tokenization
Pre Tokenization
Subtopic
5 questions
Questions tagged with Pre Tokenization — part of Tokenization.
Format
Difficulty
Role
Experience
Companies
Sort
Newest
Quality
Difficulty ↑
Difficulty ↓
Questions
Pre-tokenization splits text before BPE merges. Why is that boundary important?
Flashcard
Easy
·
Qual 4.0
How would you train a custom tokenizer for a genomics LLM on DNA sequences?
Short Answer
Hard
·
Qual 4.0
What happens to emoji ZWJ sequences under cl100k_base vs. o200k_base pre-tokenization?
Short Answer
Hard
·
Qual 4.0
OpenAI
What is the primary purpose of regex-based pre-tokenization in cl100k_base, and what constraint does it impose on BPE?
Multiple Choice
Medium
·
Qual 4.0
What constraint does BoundlessBPE relax, and what improvement does it yield?
Flashcard
Hard
·
Qual 4.0