PremiumWhat does the '200k'…
Multiple ChoiceMediumAi4bharatWorkday
Unlock with Pro to study this question. See Pro pricing.
The comparison table for this question.
Explain the role of regex based pre-tokenization in cl100k_base (GPT-4) and o200k_base (GPT-4o). Focus specifically on how each handles Unicode ZWJ (Zero Width Joiner) emoji sequences, and why the o200k_base regex was updated.
| Aspect | cl100k_base (GPT-4) | o200k_base (GPT-4o) |
|---|---|---|
| ZWJ emoji sequence | One chunk (symbol branch; no ZWJ rule) | Usually one chunk; split after U+FE0F, which counts as a mark |
| Tokens for the rainbow flag 🏳️🌈 | 9 (few emoji merges) | 6 (more byte merges) |
| Emoji modifiers (skin tone, flags) | One chunk, many byte-level tokens | One chunk, fewer tokens |
| Multilingual character runs | Breaks at combining marks (Devanagari vowel signs) | Keeps combining marks with their letters |
| Design motivation | ASCII-tuned | Multilingual and multimodal |
Morphologically rich word forms (verb conjugations, case endings) and accented characters that have fewer learned merge rules than English patterns.
European languages use longer sentences grammatically, so more tokens are needed to express the same concept.
The BPE tokenizer assigns special prefix tokens to each non-English word, doubling the token count.
European languages use Unicode codepoints above U+007F, which always encode as two separate tokens in BPE.
| Cause | Effect on European words | Why |
|---|---|---|
| Accented characters (é, ü, ñ) | Fewer single-token forms | Rare in English-dominant training data |
| Rich morphology (-ción, -heit) | Words split into stem plus ending | Each word form is rarer than its English counterpart |
| Sparse merge coverage | Words rebuilt from subunits | Vocabulary budget spent on English |
| Net result | Higher fertility, higher cost | Cost scales with token count |
Using tiktoken with cl100k_base (GPT-4's tokenizer), compare the token count for:
Arabic characters use 2 bytes each in UTF-8. The cl100k_base tokenizer was trained predominantly on English data. Calculate the token count for each string and the approximate ratio.
| Property | English 'Hello world' | Arabic 'مرحبا بالعالم' |
|---|---|---|
| UTF-8 bytes per character | 1 byte (ASCII) | 2 bytes |
| Learned merge coverage | Extensive (word-level) | Sparse (few pairs merge) |
| Approx tokens | ~2 | ~8-10 |
| Fertility (tokens/word) | ~1 | ~4-5 |
| Relative cost / context use | Baseline | 4-5x higher |
Your team's English only embedding pipeline processes 1M product descriptions averaging 150 tokens each, costing USD X at USD 0.10/1M tokens. You are expanding to Spanish and French. Describe how tokenizer fertility affects your cost estimate, quantify the expected token inflation, and outline the correct methodology for building an accurate budget before the pipeline launches.
| Scenario | Total tokens | Cost at $0.10/1M |
|---|---|---|
| English baseline (1.0x) | 150M | $15.00 |
| Spanish/French at 1.2x | 180M | $18.00 (+20%) |
| Spanish/French at 1.5x | 225M | $22.50 (+50%) |
A model has d_model=4096 and you are considering three vocabulary sizes: 8k, 50k, and 200k tokens. Calculate the bfloat16 memory footprint of the combined embedding table + lm_head for each size. Then describe the practical tradeoffs (sequence length, softmax cost, multilingual coverage) of each vocabulary size choice.
| Concern | Small vocab (8k) | Large vocab (200k) |
|---|---|---|
| Interface memory at d=4096 | ~131 MB | ~3.3 GB |
| Sequence length / fertility | High (many tokens per word) | Low (compact tokens) |
| Attention cost | Worse (longer sequences, O(n squared)) | Better (shorter sequences) |
| Output softmax cost | Cheap per token | Expensive per token |
| Multilingual coverage | Poor outside English | Much better across scripts |
| Best fit | Tiny-alphabet domains | Multilingual frontier models |
A product manager claims 'Our 128k context model can hold a 128,000 word document.' You are building a multilingual RAG pipeline serving English, French, Japanese, and Arabic users. Correct the PM's claim for English and explain how token budgeting must differ for each language. Then outline a practical method for estimating per language token budgets.
| Language | Approx. fertility (tokens/word) | Words in a 128k window |
|---|---|---|
| English | ~1.3 | ~96,000 |
| French | ~1.3-1.5 | ~85,000-98,000 |
| Japanese | ~3-5 | ~25,000-40,000 |
| Arabic | ~3-5 | ~25,000-40,000 |
A finance leader hands you a spreadsheet ranking three LLM providers by published $ per million output tokens. Explain why the ranking is misleading even when the prices are accurate, and describe the comparison you would run instead. Cover the role of tokenizer family, the typical magnitude of the gap, and the unit that matters to a buyer.
| Comparison unit | Tokenizer-independent? | What it tells you | When to use |
|---|---|---|---|
| $/M tokens (headline) | No, family-specific | Provider's pricing within its own unit | Within-family rate changes only |
| $/M characters input + $/M characters output | Yes | Honest cross-provider total cost | Procurement decisions |
| $ per representative request | Yes (corpus-fixed) | Practical real-traffic cost | Pre-purchase load testing |
| $ per delivered useful answer | Yes (semantic) | What the buyer actually cares about | Production cost dashboards |
| Model family | Encoding | How to count input |
|---|---|---|
| GPT-5.5 | No tiktoken mapping; estimate with o200k_base | tiktoken.get_encoding('o200k_base') + .encode(), then check usage.prompt_tokens |
| GPT-4 era (4o, 4-turbo) | o200k_base / cl100k_base | Resolve via encoding_for_model per specific model |
| Claude Opus 4.7 | Anthropic internal | client.messages.count_tokens(...) |
| Gemini 3.1 Pro | Google internal | model.count_tokens(...) on the Vertex SDK |
| Llama 4 / open weight | Model specific | AutoTokenizer.from_pretrained(...).encode(text) |
Compare the sentencepiece Python library (Google's original implementation) and the HuggingFace tokenizers library when used to tokenize input for a Llama 2 model. Explain when tokenization drift can occur and which library is recommended for production Llama serving.
| Property | sentencepiece (C++/Python) | HuggingFace tokenizers (Rust) |
|---|---|---|
| Loads | .model binary file | tokenizer.json + tokenizer_config.json |
| Role | Algorithmic reference / ground truth | Canonical for Hub-distributed models |
| Chat template | Not supported | apply_chat_template() built in |
| Ecosystem | Standalone | AutoTokenizer, fast path, offset mapping |
| Edge-case IDs | Reference behavior | May drift on rare Unicode / boundaries |
| Recommended for Llama serving | No | Yes |
Sparse retrieval is faster, so RRF uses sparse as the fast path and falls back to dense only on cache misses
Dense embeddings smooth over exact match signals (rare names, IDs, code symbols); sparse (BM25) catches them. RRF combines RANKS, not scores, by summing 1/(k+rank), avoiding the cross-system score calibration problem
RRF averages cosine and BM25 scores after normalizing both to [0, 1], which gives equal weight to each retriever
Sparse and dense almost always return the same documents, so RRF is mostly a redundancy mechanism for robustness
| Property | Dense (embeddings) | Sparse (BM25) |
|---|---|---|
| Signal | Distributional / semantic similarity | Exact lexical overlap weighted by idf |
| Wins on | Paraphrase, synonyms, multilingual variants | Rare identifiers, proper nouns, error codes |
| Loses on | Out of vocabulary tokens, exact ID lookup | Paraphrase with zero word overlap |
| Index | ANN index over float vectors | Inverted index over tokens |
| Score scale | Bounded cosine ([-1, 1] or [0, 1]) | Unbounded log-domain |
| Fusion role | One of two ranked lists into RRF | One of two ranked lists into RRF |
What did OpenAI replace text-embedding-ada-002 with, and what is the main difference?
| Model | Dimensions | Price per 1M tokens | MTEB lift over ada-002 | Matryoshka |
|---|---|---|---|---|
| text-embedding-ada-002 | 1536 (fixed) | $0.10 (deprecated default) | baseline | no |
| text-embedding-3-small | 1536 (truncatable) | $0.02 | +1.3 average (62.3 vs 61.0) | yes |
| text-embedding-3-large | 3072 (truncatable) | $0.13 | +3.6 average (64.6 vs 61.0) | yes |
General English RAG with strong default quality, cheap, hosted, Matryoshka truncatable
Code symbol aware retrieval over a large source code corpus
Cross-lingual retrieval across 30+ languages, including non-Latin scripts, with sparse+dense+multi-vector outputs
Production RAG where the embedding provider also offers a tightly paired cross-encoder reranker
On prem / self-hosted deployment with no API dependency, open weights, strong MTEB
OpenAI text-embedding-3-large (or 3-small for cost)
BGE-M3
Nomic Embed v1.5 or Snowflake Arctic Embed L 2.0
Cohere embed-v3 / v4 + rerank-3.5
Voyage voyage-code-3
| Vendor / Model | Best for | Hosting | Standout feature |
|---|---|---|---|
| OpenAI text-embedding-3-large | general English RAG | hosted API | Matryoshka truncation, cheap |
| Voyage voyage-code-3 | code retrieval | hosted API | +13.8% avg over OpenAI 3-large on 32 code sets |
| BGE-M3 | multilingual, hybrid | open weights | dense + sparse + multi-vector from one call |
| Cohere embed-v3 / v4 + rerank-3.5 | paired-reranker stacks | hosted API | tight embedder-reranker pairing |
| Nomic v1.5 / Arctic Embed L 2.0 | on-prem, no API | open weights | single-H100 deployable, strong MTEB |
OpenAI text-embedding-3-small
Voyage voyage-code-3
BGE-M3
Cohere embed-v3 / v4
Nomic Embed v1.5
Open weights, Matryoshka, designed for on prem and self-hosted deployments
Open weights, multilingual (100+ langs), hybrid dense+sparse+multi-vector outputs
Specialized for code retrieval, about 14% above OpenAI v3-large on average across 32 code retrieval datasets
Tightly paired with Cohere rerank-3.5 cross-encoder; popular for production RAG stacks
$0.02/M tokens, 1536 dim (truncatable), default RAG choice
| Vendor / Model | Positioning | Hosting | Pricing or feature anchor |
|---|---|---|---|
| OpenAI text-embedding-3-small | default general English | API only | $0.02 / M tokens, 1536-d Matryoshka |
| Voyage voyage-code-3 | code retrieval specialist | API only | +13.8% avg over OpenAI 3-large on 32 code sets |
| BGE-M3 | open-weights multilingual | self-hosted | 100+ langs, dense+sparse+multi-vector |
| Cohere embed-v3 / v4 | paired-reranker stack | API only | tight pairing with rerank-3.5 cross-encoder |
| Nomic Embed v1.5 | open-weights on-prem | self-hosted | Matryoshka, single-H100 deployable |
OpenAI's batch embedding API and similar offerings from other providers discount embedding cost by ~50% in exchange for a relaxed (e.g. 24 hour) latency SLA. Explain the underlying economic reason and identify the workloads where the discount matters vs where it's irrelevant.
| Dimension | Synchronous API | Batch API |
|---|---|---|
| Latency SLA | milliseconds to seconds | up to 24 hours |
| Price | list price | ~50% of list |
| Best for | live query, realtime ingest | corpus backfill, migration, eval |
| Failure handling | retry inline | job-level retry, async callback |
| Concurrency cap | rate limits per minute | job-size cap, concurrent jobs cap |
Increase the embedding dimension. Quality always scales with d
Try a domain specific embedding model (e.g. voyage-code-3 for code, FinBERT or Voyage-finance for finance, MedCPT for biomedical) before reaching for fine-tuning, then fall back to fine-tuning if no off the shelf model fits
Switch to a different vector database. The index is probably the bottleneck
Pre-translate all documents into a canonical 'simple English' form before embedding
| Intervention | Cost | Typical lift | When to use |
|---|---|---|---|
| Domain-specific off the shelf embedder | low | 6-14% average lift on target domain (Voyage figures) | first move when general fails |
| Cross-encoder reranker | low | 5-15% nDCG lift | when no specialist exists or as add-on |
| Hybrid with BM25 | low-medium | 5-20% on rare-term domains | domains with many unique tokens |
| Fine-tune general embedder | high | 10-25% with good labeled data | after others have been tried |
You're shipping a RAG system and the off the shelf embedding model (text-embedding-3-large) underperforms expectations on your domain. Outline when fine-tuning the embedding model is the right next step vs alternatives.
| Approach | Compute cost | Operational cost | Typical lift |
|---|---|---|---|
| Off-the-shelf specialist | zero (use existing) | low (re-embed only) | 10-30% on target domain |
| Cross-encoder reranker | low (inference only) | low (deploy second model) | 5-15% nDCG |
| Hybrid retrieval (BM25) | low (BM25 is cheap) | low (second index) | 5-20% on rare-term domains |
| LoRA fine-tune | medium (one GPU-day) | high (full re-embed) | 10-20% over base |
| Full fine-tune of open base | high (multi-GPU-day) | high (full re-embed) | 10-25% over base |