Your indexing pipeline tokenizes 10M docs in 18 hours. What single change gives the biggest speedup?

MCQ·Medium·4.0 · 0·~1 min·Asked atBaidu Niki Ai Sap·Relevant atDatabricks Meta

Attempt it

The right answer is **A: switch to the fast Rust-backed tokenizer with batch encoding**. HuggingFace ships two tokenizer implementations for most models: a Python reference (slow) and a Rust-backed version (fast). The fast version is built on the `tokenizers` library and runs the BPE merge logic in Rust with internal parallelism across CPU cores. Loading the fast version is one flag: `AutoTokenizer.from_pretrained(checkpoint, use_fast=True)` (this is the default for most models in 2026). The second half of the speedup is batch encoding. Instead of calling `tokenizer(text)` in a Python loop, you call `tokenizer(list_of_texts, padding=False, truncation=True)` once with the whole batch. This amortizes the Python-to-Rust call overhead and lets the Rust side parallelize across the batch. Typical speedup combining both: 20 to 100x on a workload like this. An 18-hour job often drops to 15-60 minutes. **Why the other options are wrong.** **B (GPU):** Tokenization is CPU-bound regex and BPE work, not the kind of matrix math GPUs accelerate. Moving to GPU adds transfer overhead and no compute benefit. **C (higher max_length):** Truncation is not the bottleneck. max_length controls how many tokens to keep, not how fast the tokenizer runs. **D (cache to disk):** Useful as a secondary optimization for reruns, but does not speed up the first run and does not fix the underlying slow loop.

**A: fast Rust-backed tokenizer + batch encoding.** HuggingFace ships two tokenizer implementations per model: a Python reference and a Rust-backed fast tokenizer (the `tokenizers` library). The fast version is the default for most models in 2026 (`use_fast=True`). It runs the BPE merge logic in Rust with internal CPU-core parallelism, and exposes a `__call__` that accepts either a single string or a list. The combined fix: `tokenizer(list_of_texts, padding=False, truncation=True, return_tensors=None)`. Batch encoding amortizes the Python-to-Rust boundary call overhead, and the Rust side parallelizes across cores. Typical speedup on this workload is 20-100x; an 18-hour job often drops to 15-60 minutes. **Secondary wins** worth knowing. - Drop padding and tensor conversion at tokenize time. Pad only at training/inference batch time inside the collator. - For multi-machine fan-out, use a process pool (joblib, ray, multiprocessing) to split the document set across machines; each machine still uses the fast tokenizer plus batch encoding. - For OpenAI-compatible workloads, `tiktoken` is another Rust-backed option with similar performance characteristics. **Why the wrong options are wrong.** **B (GPU):** Tokenization is byte-level regex split plus BPE merge lookup. Both are pointer-chasing CPU work, not tensor math. GPU adds nothing and adds transfer cost. **C (higher max_length):** Affects how much you keep per document, not throughput. **D (cache to disk):** Useful for re-runs (the second time the same corpus is tokenized) but does nothing for the first run. Also adds disk-write throughput as a new bottleneck.

TL;DR

Use the Rust-backed fast tokenizer with batch encoding (tokenizer(list_of_texts)) instead of a per-doc Python loop. CPU-parallel internally, FFI overhead amortized. Typical 20-100x speedup.

Memory aid

Easy to grasp

Imagine you have a million letters to stamp. The slow way is to walk one letter to the stamping desk, stamp it, walk back, pick up the next one. The fast way is to dump all million letters on the desk and use a machine that stamps them in parallel. The slow Python tokenizer takes a one at a time approach. The Rust fast tokenizer is the parallel-stamping machine. Batch encoding is dumping all the letters at once instead of one at a time. Same model, same output, dramatically less wasted time at the door of the stamping room. Moving to a GPU is wrong because the stamping machine is a CPU machine. Caching the stamped letters helps next time but does not fix the slow loop now. Setting a higher length cap changes how you cut the letters, not how fast you stamp them.

Key concepts

Fast tokenizer Batch encoding HuggingFace tokenizers

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

An 18-hour tokenization job on 10M documents is one of the most common performance complaints in data engineering for RAG and pretraining pipelines. The instinct is to reach for the usual optimization toolkit: parallelize on GPU, increase batch size, reduce the input. All three are wrong on this workload, and the right fix is a much smaller code change with much larger impact.

The correct framing is that tokenization on the slow Python path is bottlenecked by two layers of overhead: the per-document Python-function call overhead, and the per-step Python-to-tokenizer dispatch. The fast Rust-backed tokenizer addresses the second; batch encoding addresses the first. Together they yield a typical 20-100x speedup, turning an 18-hour job into a 15-60 minute job.

The rest of this explanation walks why tokenization is structurally CPU-bound, what the fast tokenizer does internally, why batch encoding matters separately from the fast tokenizer, what the per-machine and multi-machine scaling look like, and why each of the three wrong options misdiagnoses the bottleneck.

Why tokenization is CPU-bound and Python overhead-bound

BPE tokenization is two layers of work. The first is pre-tokenization: applying a regex (Unicode-aware in modern tokenizers) to split the input text into chunks like words, punctuation, and whitespace. The second is merge lookup: for each chunk, repeatedly looking up adjacent byte pairs in the BPE merge table and replacing them with the merged token until no more merges apply.

Both steps are pointer-heavy, branch-heavy, and memory-irregular. The pre-tokenization regex involves Unicode property tables; the merge lookups involve hash-table operations on small strings or byte sequences. Neither is the kind of dense numeric work that benefits from SIMD or GPU acceleration. The bottleneck is single-thread Python execution speed.

On top of the algorithmic cost, the slow Python tokenizer pays interpreter overhead at every level: per-character iteration in the pre-tokenizer, per-pair lookup in the merge step, per-document dispatch from the calling code. For a workload of 10M documents, the cumulative interpreter overhead can be 10-20x the actual algorithmic work.

The fast Rust-backed implementation eliminates the interpreter overhead by running everything in compiled Rust code with internal CPU-core parallelism (typically via Rayon, the Rust parallelism library). On a 32-core machine, a single batch call can saturate all cores with embarrassingly-parallel per-document tokenization. The speedup over single-thread Python is a function of both the constant-factor compilation win and the core-count parallelism win.

What the fast tokenizer gives you and how to load it

Why batch encoding is the second half of the fix

Why the three wrong options are wrong

Multi-machine fan-out and the secondary optimizations

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

from transformers import AutoTokenizer

# Slow: per-document loop with slow tokenizer
tok_slow = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B', use_fast=False)
results_slow = [tok_slow.encode(doc) for doc in docs]  # 18 hours on 10M docs

# Fast: Rust tokenizer + batch encoding
tok_fast = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B', use_fast=True)
results_fast = tok_fast(
    docs,  # full list, not loop
    padding=False,
    truncation=True,
    max_length=2048,
    return_tensors=None,
)['input_ids']
# 15-60 minutes on 10M docs, 20-100x speedup

# Even better for very large corpora: batch in chunks of 1K-10K
def batched(seq, n):
    for i in range(0, len(seq), n):
        yield seq[i:i+n]
results = []
for batch in batched(docs, 5000):
    results.extend(tok_fast(batch, padding=False, truncation=True)['input_ids'])

Real products, models, and research that use this idea.

HuggingFace tokenizers library ships Rust-backed fast tokenizers for Llama 3+, Mistral, Qwen 3.5, and most modern open-weight models; use_fast=True is the default in transformers 4.40+.
tiktoken (OpenAI's Rust-backed BPE library) achieves similar throughput characteristics for cl100k_base and o200k_base workloads.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhen does the fast tokenizer NOT provide a 20-100x speedup?

When the documents are very long (the Rust merge work dominates the FFI overhead, so batching helps less), when the model only ships a slow tokenizer (rare in 2026 but happens with research models), when downstream code immediately blocks on each result (preventing parallelism), or when the batch sizes are tiny (1-10 documents per call, where overhead amortization is poor). Profile to confirm where the bottleneck actually lives.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming tokenization performance is a GPU problem or an algorithm problem, when the real issue is Python call overhead on a slow per-document loop.

60 second bullets to scan on the way to the call.

Why tokenization is CPU-bound and GPU acceleration does not help.
Difference between the slow Python tokenizer and the fast Rust-backed tokenizer.

Primary sources. Browse if you want the original framing.