$50K compute budget, 50B tokens of US legal text, 1.5B parameter model. Defend your tokenizer design.

Short answer·Hard·4.0 · 0·~3 min·Asked atAnthropic Meta OpenAI

Attempt it

Free · 2 AI evals / day

**Algorithm: byte-level BPE.** Fast to train, zero OOV via byte fallback, well-supported in inference stacks. Unigram-LM is the research-grade alternative but doubles tokenizer-training compute; not worth it on this budget. **Vocab size: 48K-64K.** For a 1.5B model with d_model=2048, a 64K vocab adds 64K * 2048 * 2 = ~262M parameters in embedding+lm_head combined. That is ~17 percent of the 1.5B parameter budget. Above 64K, parameters bloat without proportional fertility gain on legal text. Below 32K, common multi-word legal terms ('habeas corpus', 'force majeure', '17 C.F.R.') fragment, inflating sequence length. **Special tokens.** Standard: BOS, EOS, PAD. Domain-specific: [CITATION] for case citations like 'Brown v. Board, 347 U.S. 483', [STATUTE] for statute references, [SECTION] for section markers in SEC filings, [PARTY] for party names in contracts. These let the model attend to document structure via single tokens rather than learning the structure from raw punctuation. **Training strategy.** Train the BPE on a stratified 5GB sample, NOT the full 50B tokens. BPE diminishing returns kick in fast; top merges stabilize within a few GB on a domain corpus. Save the compute for pretraining. **Validation BEFORE committing the $50K.** Run the proposed tokenizer on a held-out 1GB legal sample. Compare fertility against cl100k_base on the same sample. The domain BPE should save 15-25 percent in sequence length. If it saves less than 10 percent, the vocab choice is wrong, reconsider before pretraining. **Budget split.** ~70 percent for pretraining (large-batch training the 1.5B on 50B tokens). ~30 percent for inference experiments: prompt engineering, RAG, downstream task evaluation.

**Algorithm: byte-level BPE.** Production default. Fast to train (~hours for the 5GB sample), zero OOV via byte fallback, well-supported in HuggingFace tokenizers and tiktoken stacks. Unigram-LM doubles training compute; not justified on this budget. **Vocab size: 48K-64K.** Parameter budget calculation. For 1.5B with d_model=2048, embedding+lm_head = vocab_size * 2048 * 2 params. 32K = 131M (~9 percent). 64K = 262M (~17 percent). 128K = 524M (~35 percent of model, untenable for 1.5B). For legal English, optimum is 48K-64K. Below 32K, multi-word legal terms ('habeas corpus', 'force majeure', '17 C.F.R.') fragment to 3-5 tokens, inflating sequence length. Above 64K, fertility plateaus. Pick 64K for safety; 48K if memory is tight. **Special tokens.** Standard: BOS, EOS, PAD. Domain structural tokens via preprocessing: [CITATION], [STATUTE], [SECTION], [PARTY], [EXHIBIT]. Preprocessing regex finds recognized patterns and wraps them, so single tokens at known positions let the model attend to document structure cheaply. **Training strategy.** Train BPE on stratified 5GB sample, NOT the full 50B. BPE has diminishing returns past a few GB on a homogeneous corpus. Stratify across SEC filings, federal case law, state case law, regulations, contracts. Tokenizer training takes ~hours, tiny fraction of budget. **Validation before committing.** Take 1GB held-out legal sample. Tokenize with proposed vocab AND cl100k_base. The domain BPE should save 15-25 percent in sequence length. Below 10 percent saving, the design is wrong; reconsider before spending $50K. A 20 percent fertility win means 10B effective tokens saved across the 50B run, justifying the 262M parameter cost. **Budget split: 70/30.** ~70 percent pretraining (optimizer states, gradients, multi-epoch). ~30 percent inference experiments: prompt engineering, RAG, downstream task fine-tuning.

TL;DR

Byte-level BPE, 48K-64K vocab (~262M params at d_model=2048), plus structural [CITATION]/[STATUTE]/[SECTION]/[PARTY]. Train BPE on 5GB stratified sample. Validate fertility vs cl100k_base on 1GB before committing.

Memory aid

Easy to grasp

Imagine you have a budget to teach a small student a specialized job, reading legal contracts. You decide what shorthand vocabulary they will learn first. Too few shortcuts (only the alphabet), and every common term like 'force majeure' takes many slow letters to read. Too many shortcuts, and the student spends most of their brain power memorizing the shorthand instead of learning to think. The sweet spot is roughly 50,000 shortcuts. Of those, a few are special markers like 'this is a case citation' or 'this is a statute number' that help the student navigate the document structure. Before you spend the whole budget, you spot-check: take a sample of contracts, count how much faster the new shorthand reads them versus the off the shelf shorthand a general-purpose student would use. If your shorthand saves 20 percent of reading time, the budget is worth spending.

Key concepts

Domain-specific BPE Vocab size Structural special tokens Compute budget

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Designing a domain-specific tokenizer on a constrained compute budget is one of the most concrete budget versus quality tradeoffs in LLM engineering. Every design choice has a measurable cost (parameters, training compute, engineering time) and a measurable benefit (sequence length saved, downstream quality improved, structural reasoning enabled). The discipline is to compute both before spending, validate empirically on a small held-out sample, and only commit the larger compute once the design has cleared the gating experiment.

The correct answer for this scenario has five components: byte-level BPE algorithm, 48K-64K vocab justified by parameter math at d_model=2048, BOS/EOS/PAD plus domain structural tokens, stratified 5GB tokenizer-training sample, and a validation experiment against cl100k_base on 1GB held-out before committing the pretraining compute. The budget split is roughly 70/30 between pretraining and inference experiments.

The rest of this explanation walks each design choice with the explicit parameter and compute math, names the production examples (BloombergGPT, Galactica, LegalBERT), and closes with the discipline of validating before committing.

Algorithm: why byte-level BPE on this budget

Byte-level BPE is the production default in 2026 for several reasons. Fast to train: a few hours on a single machine for a 5GB sample, versus days for Unigram-LM. Zero OOV via byte fallback: any UTF-8 input encodes, even if it consists entirely of bytes the BPE never saw. Well-supported in HuggingFace tokenizers (Rust-backed) and tiktoken-style inference libraries: integration with production serving stacks is a one-line config change. Fast at inference: the Rust implementation handles batch encoding at high throughput.

Unigram-LM with subword regularization is the research-grade alternative. It has a more principled probabilistic framework, naturally produces multiple valid tokenizations of the same input (useful for robustness training), and handles morphologically rich languages more elegantly. The downsides: tokenizer training takes 2-3x longer, inference is slower, the ecosystem is smaller. For a $50K budget on a legal-domain model, byte-level BPE wins.

SentencePiece is another option to consider; it is essentially BPE or Unigram-LM with a unified interface for handling Unicode normalization and word boundaries. For Latin-script domains like legal English, byte-level BPE via HuggingFace tokenizers is simpler and equally effective.

The algorithm choice is rarely the most consequential decision on this budget. Vocab size and the validation experiment matter more. Pick byte-level BPE and move on.

Vocab size: explicit parameter math

Special tokens: standard plus domain structural

Training strategy: stratified 5GB sample

Validation: the gating experiment before commit

Budget split and the production discipline

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders

# 1. Initialize byte-level BPE
tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel()
tok.decoder = decoders.ByteLevel()

# 2. Configure trainer with domain structural tokens
trainer = trainers.BpeTrainer(
    vocab_size=64000,
    special_tokens=['<|bos|>', '<|eos|>', '<|pad|>',
                    '[CITATION]', '[STATUTE]', '[SECTION]', '[PARTY]', '[EXHIBIT]'],
    min_frequency=5,
)

# 3. Train on stratified 5GB sample (NOT the full 50B)
tok.train(['stratified_legal_5gb.txt'], trainer)
tok.save('legal_bpe_64k.json')

# 4. Validate fertility vs cl100k_base BEFORE pretraining
import tiktoken
enc = tiktoken.get_encoding('cl100k_base')
held_out = open('legal_holdout_1gb.txt').read()

legal_tokens = tok.encode(held_out).ids
general_tokens = enc.encode(held_out)

ratio = len(legal_tokens) / len(general_tokens)
print(f'Legal BPE produces {ratio:.2%} of cl100k_base length')
# Expect 0.75-0.85 (15-25% shorter). Below 0.90, reconsider design.

Real products, models, and research that use this idea.

LegalBERT and Legal-RoBERTa (2020-2022 era) used domain-specific tokenizers tuned for case law, demonstrating the sequence-length wins on legal corpora.
BloombergGPT (2023) trained a domain-specific 50K BPE for financial text, showing the same domain-tokenizer pattern at scale.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the calculus change if the budget were $500K instead of $50K?

Larger budget allows: bigger model (7B not 1.5B), larger vocab (96K-128K becomes proportional), more rigorous tokenizer experiments (multiple candidate vocabs, ablation studies, Unigram-LM as a comparison). The 70/30 pretraining/inference split might shift toward 60/40 because the inference experiments have more bandwidth. The fundamental design principles do not change, but the resolution at which you can validate them improves.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Choosing a vocab size based on what other models use (32K Llama 2, 128K Llama 3) without computing the embedding parameter cost as a fraction of the target model size.

60 second bullets to scan on the way to the call.

Why byte-level BPE is the right algorithm for a budget-constrained domain model.
How to compute embedding+lm_head parameter cost as vocab_size * d_model * 2.

Primary sources. Browse if you want the original framing.