$50K compute budget, 50B tokens of US legal text, 1.5B parameter model. Defend your tokenizer design.

Short answer·Hard·4.0 · 0·~3 min·Relevant atAnthropic Meta OpenAI

Attempt it

Free · 2 AI evals / day

**Algorithm: byte-level BPE.** Fast to train, zero OOV via byte fallback, well-supported in inference stacks. Unigram-LM is the research-grade alternative but doubles tokenizer-training compute; not worth it on this budget. **Vocab size: 48K-64K.** For a 1.5B model with d_model=2048, a 64K vocab adds 64K * 2048 * 2 = ~262M parameters in embedding+lm_head combined. That is ~17 percent of the 1.5B parameter budget. Above 64K, parameters bloat without proportional fertility gain on legal text. Below 32K, common multi-word legal terms ('habeas corpus', 'force majeure', '17 C.F.R.') fragment, inflating sequence length. **Special tokens.** Standard: BOS, EOS, PAD. Domain-specific: [CITATION] for case citations like 'Brown v. Board, 347 U.S. 483', [STATUTE] for statute references, [SECTION] for section markers in SEC filings, [PARTY] for party names in contracts. These let the model attend to document structure via single tokens rather than learning the structure from raw punctuation. **Training strategy.** Train the BPE on a stratified 5GB sample, NOT the full 50B tokens. BPE diminishing returns kick in fast; top merges stabilize within a few GB on a domain corpus. Save the compute for pretraining. **Validation BEFORE committing the $50K.** Run the proposed tokenizer on a held-out 1GB legal sample. Compare fertility against cl100k_base on the same sample. The domain BPE should save 15-25 percent in sequence length. If it saves less than 10 percent, the vocab choice is wrong, reconsider before pretraining. **Budget split.** ~70 percent for pretraining (large-batch training the 1.5B on 50B tokens). ~30 percent for inference experiments: prompt engineering, RAG, downstream task evaluation.

**Algorithm: byte-level BPE.** Production default. Fast to train (~hours for the 5GB sample), zero OOV via byte fallback, well-supported in HuggingFace tokenizers and tiktoken stacks. Unigram-LM doubles training compute; not justified on this budget. **Vocab size: 48K-64K.** Parameter budget calculation. For 1.5B with d_model=2048, embedding+lm_head = vocab_size * 2048 * 2 params. 32K = 131M (~9 percent). 64K = 262M (~17 percent). 128K = 524M (~35 percent of model, untenable for 1.5B). For legal English, optimum is 48K-64K. Below 32K, multi-word legal terms ('habeas corpus', 'force majeure', '17 C.F.R.') fragment to 3-5 tokens, inflating sequence length. Above 64K, fertility plateaus. Pick 64K for safety; 48K if memory is tight. **Special tokens.** Standard: BOS, EOS, PAD. Domain structural tokens via preprocessing: [CITATION], [STATUTE], [SECTION], [PARTY], [EXHIBIT]. Preprocessing regex finds recognized patterns and wraps them, so single tokens at known positions let the model attend to document structure cheaply. **Training strategy.** Train BPE on stratified 5GB sample, NOT the full 50B. BPE has diminishing returns past a few GB on a homogeneous corpus. Stratify across SEC filings, federal case law, state case law, regulations, contracts. Tokenizer training takes ~hours, tiny fraction of budget. **Validation before committing.** Take 1GB held-out legal sample. Tokenize with proposed vocab AND cl100k_base. The domain BPE should save 15-25 percent in sequence length. Below 10 percent saving, the design is wrong; reconsider before spending $50K. A 20 percent fertility win means 10B effective tokens saved across the 50B run, justifying the 262M parameter cost. **Budget split: 70/30.** ~70 percent pretraining (optimizer states, gradients, multi-epoch). ~30 percent inference experiments: prompt engineering, RAG, downstream task fine-tuning.

Quick reference

TL;DR

Byte-level BPE, 48K-64K vocab (~262M params at d_model=2048), plus structural [CITATION]/[STATUTE]/[SECTION]/[PARTY]. Train BPE on 5GB stratified sample. Validate fertility vs cl100k_base on 1GB before committing.

Memory aid

Easy to grasp

Imagine you have a budget to teach a small student a specialized job, reading legal contracts. You decide what shorthand vocabulary they will learn first. Too few shortcuts (only the alphabet), and every common term like 'force majeure' takes many slow letters to read. Too many shortcuts, and the student spends most of their brain power memorizing the shorthand instead of learning to think. The sweet spot is roughly 50,000 shortcuts. Of those, a few are special markers like 'this is a case citation' or 'this is a statute number' that help the student navigate the document structure. Before you spend the whole budget, you spot-check: take a sample of contracts, count how much faster the new shorthand reads them versus the off the shelf shorthand a general-purpose student would use. If your shorthand saves 20 percent of reading time, the budget is worth spending.

Key concepts

Domain-specific BPE Vocab size Structural special tokens Compute budget

Detailed answer & concept explanation

~10 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Open with the framing that tokenizer design on a constrained budget is parameter-budget plus sequence-length budget optimization with validation before commit. Walk the five axes: byte-level BPE, 48K-64K vocab with explicit param math at d_model=2048, BOS/EOS/PAD plus structural [CITATION]/[STATUTE]/[SECTION]/[PARTY], stratified 5GB training sample, validation against cl100k_base on a 1GB held-out. Cover the budget split (70 percent pretraining / 30 percent inference experiments). Close with the discipline: validate the fertility delta empirically before committing the $50K, and only commit if the saving compounds favorably across 50B pretraining tokens.

Code example

python

from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders

# 1. Initialize byte-level BPE
tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel()
tok.decoder = decoders.ByteLevel()

# 2. Configure trainer with domain structural tokens
trainer = trainers.BpeTrainer(
    vocab_size=64000,
    special_tokens=['<|bos|>', '<|eos|>', '<|pad|>',
                    '[CITATION]', '[STATUTE]', '[SECTION]', '[PARTY]', '[EXHIBIT]'],
    min_frequency=5,
)

# 3. Train on stratified 5GB sample (NOT the full 50B)
tok.train(['stratified_legal_5gb.txt'], trainer)
tok.save('legal_bpe_64k.json')

# 4. Validate fertility vs cl100k_base BEFORE pretraining
import tiktoken
enc = tiktoken.get_encoding('cl100k_base')
held_out = open('legal_holdout_1gb.txt').read()

legal_tokens = tok.encode(held_out).ids
general_tokens = enc.encode(held_out)

ratio = len(legal_tokens) / len(general_tokens)
print(f'Legal BPE produces {ratio:.2%} of cl100k_base length')
# Expect 0.75-0.85 (15-25% shorter). Below 0.90, reconsider design.

Real-world examples

Real products, models, and research that use this idea.

LegalBERT and Legal-RoBERTa (2020-2022 era) used domain-specific tokenizers tuned for case law, demonstrating the sequence-length wins on legal corpora.
BloombergGPT (2023) trained a domain-specific 50K BPE for financial text, showing the same domain-tokenizer pattern at scale.
Galactica (Meta, 2022) used a domain-specific tokenizer for scientific text with special tokens for citations, math, and chemical formulas.
Code Llama and DeepSeek-Coder use code-aware BPE tokenizers; the legal-domain design parallels their approach but for a different specialty.
HuggingFace tokenizers and tiktoken-style libraries both support byte-level BPE training on domain corpora with a few lines of code, making the engineering investment small.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the calculus change if the budget were $500K instead of $50K?

QWhy insert structural tokens via preprocessing rather than learning them from raw text via BPE merges?

QIf your validation experiment showed the domain BPE saves only 8 percent versus cl100k_base, what would you do?

QHow would you justify the 64K vocab choice if a stakeholder pushes for 128K because 'Llama 3 uses 128K'?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Choosing a vocab size based on what other models use (32K Llama 2, 128K Llama 3) without computing the embedding parameter cost as a fraction of the target model size.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.