Part ofFresher / New Grad·Week 1: Core Concepts·question 5 of 6View roadmap →

Why does BPE tokenization use subwords instead of words or characters?

Flashcard·Easy·4.4 · 62·~30s·Relevant atOpenAI

Next in this stepContinue

Similar questions

Same topic, related formats. Practice these next.

4 curated

Why must developers use a model's specific apply_chat_template() rather than building the prompt string manually?

MCQ·Medium·4.0 ★

BPE learned 50,000 merge rules during training. What is a merge rule, and where does it live at inference?

Flashcard·Easy·4.0 ★

You are building a 7B model for 25 languages including Hindi, Arabic, and Swahili. Design a fair tokenizer and name what it costs.

Short answer·Hard·4.0 ★

Which tokenizer parameter prevents user text from being interpreted as special tokens?

MCQ·Medium·4.0 ★

Next question

Predict the output: how many tokens does this string become with a typical BPE tokenizer?

Predict output·Easy

import tiktoken # cl100k_base is the BPE vocab used by GPT-4 family models enc = tiktoken.get_encoding("cl100k_base") text = "Tokenization is the first layer of every LLM." ids = enc.encode(text) print(len(text), "chars ->", len(ids), "tokens") print([enc.decode([i]) for i in ids]) # Common words stay whole, rare patterns split. # 'Tokenization' is one token; an unseen word like 'flibbertigibbet' # would split into several known subwords.