$50K compute budget, 50B tokens of US legal text, 1.5B parameter model. Defend your tokenizer design.
Free · 2 AI evals / day
**Algorithm: byte-level BPE.** Fast to train, zero OOV via byte fallback, well-supported in inference stacks. Unigram-LM is the research-grade alternative but doubles tokenizer-training compute; not worth it on this budget.
**Vocab size: 48K-64K.** For a 1.5B model with d_model=2048, a 64K vocab adds 64K * 2048 * 2 = ~262M parameters in embedding+lm_head combined. That is ~17 percent of the 1.5B parameter budget. Above 64K, parameters bloat without proportional fertility gain on legal text. Below 32K, common multi-word legal terms ('habeas corpus', 'force majeure', '17 C.F.R.') fragment, inflating sequence length.
**Special tokens.** Standard: BOS, EOS, PAD. Domain-specific: [CITATION] for case citations like 'Brown v. Board, 347 U.S. 483', [STATUTE] for statute references, [SECTION] for section markers in SEC filings, [PARTY] for party names in contracts. These let the model attend to document structure via single tokens rather than learning the structure from raw punctuation.
**Training strategy.** Train the BPE on a stratified 5GB sample, NOT the full 50B tokens. BPE diminishing returns kick in fast; top merges stabilize within a few GB on a domain corpus. Save the compute for pretraining.
**Validation BEFORE committing the $50K.** Run the proposed tokenizer on a held-out 1GB legal sample. Compare fertility against cl100k_base on the same sample. The domain BPE should save 15-25 percent in sequence length. If it saves less than 10 percent, the vocab choice is wrong, reconsider before pretraining.
**Budget split.** ~70 percent for pretraining (large-batch training the 1.5B on 50B tokens). ~30 percent for inference experiments: prompt engineering, RAG, downstream task evaluation.
**Algorithm: byte-level BPE.** Production default. Fast to train (~hours for the 5GB sample), zero OOV via byte fallback, well-supported in HuggingFace tokenizers and tiktoken stacks. Unigram-LM doubles training compute; not justified on this budget.
**Vocab size: 48K-64K.** Parameter budget calculation. For 1.5B with d_model=2048, embedding+lm_head = vocab_size * 2048 * 2 params. 32K = 131M (~9 percent). 64K = 262M (~17 percent). 128K = 524M (~35 percent of model, untenable for 1.5B).
For legal English, optimum is 48K-64K. Below 32K, multi-word legal terms ('habeas corpus', 'force majeure', '17 C.F.R.') fragment to 3-5 tokens, inflating sequence length. Above 64K, fertility plateaus. Pick 64K for safety; 48K if memory is tight.
**Special tokens.** Standard: BOS, EOS, PAD. Domain structural tokens via preprocessing: [CITATION], [STATUTE], [SECTION], [PARTY], [EXHIBIT]. Preprocessing regex finds recognized patterns and wraps them, so single tokens at known positions let the model attend to document structure cheaply.
**Training strategy.** Train BPE on stratified 5GB sample, NOT the full 50B. BPE has diminishing returns past a few GB on a homogeneous corpus. Stratify across SEC filings, federal case law, state case law, regulations, contracts. Tokenizer training takes ~hours, tiny fraction of budget.
**Validation before committing.** Take 1GB held-out legal sample. Tokenize with proposed vocab AND cl100k_base. The domain BPE should save 15-25 percent in sequence length. Below 10 percent saving, the design is wrong; reconsider before spending $50K. A 20 percent fertility win means 10B effective tokens saved across the 50B run, justifying the 262M parameter cost.
**Budget split: 70/30.** ~70 percent pretraining (optimizer states, gradients, multi-epoch). ~30 percent inference experiments: prompt engineering, RAG, downstream task fine-tuning.
Quick reference
TL;DR
Byte-level BPE, 48K-64K vocab (~262M params at d_model=2048), plus structural [CITATION]/[STATUTE]/[SECTION]/[PARTY]. Train BPE on 5GB stratified sample. Validate fertility vs cl100k_base on 1GB before committing.
Imagine you have a budget to teach a small student a specialized job, reading legal contracts. You decide what shorthand vocabulary they will learn first. Too few shortcuts (only the alphabet), and every common term like 'force majeure' takes many slow letters to read. Too many shortcuts, and the student spends most of their brain power memorizing the shorthand instead of learning to think. The sweet spot is roughly 50,000 shortcuts. Of those, a few are special markers like 'this is a case citation' or 'this is a statute number' that help the student navigate the document structure. Before you spend the whole budget, you spot-check: take a sample of contracts, count how much faster the new shorthand reads them versus the off the shelf shorthand a general-purpose student would use. If your shorthand saves 20 percent of reading time, the budget is worth spending.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A domain where general-purpose tokenizers (cl100k_base, o200k_base) already perform well, such as conversational English. The fertility delta is small, so a domain BPE earns less than its parameter cost. Use the off the shelf tokenizer and spend the compute elsewhere.
A model class where vocab size is dwarfed by other parameters (very large models above 70B). On a 70B model, a 64K vocab is less than 1 percent of total params, so the parameter cost is negligible and the budget calculus is different; vocab choice can be larger and more generous.
An adaptation of a pretrained instruction-tuned model rather than from-scratch pretraining. Tokenizer reuse is required when fine-tuning an instruction-tuned base (covered in a sibling angle); domain-specific tokenizer design only applies when you are pretraining or near-pretraining.
Quick scan · 2–4 min
Everything important, quickly.
Open with the framing that tokenizer design on a constrained budget is parameter-budget plus sequence-length budget optimization with validation before commit. Walk the five axes: byte-level BPE, 48K-64K vocab with explicit param math at d_model=2048, BOS/EOS/PAD plus structural [CITATION]/[STATUTE]/[SECTION]/[PARTY], stratified 5GB training sample, validation against cl100k_base on a 1GB held-out. Cover the budget split (70 percent pretraining / 30 percent inference experiments). Close with the discipline: validate the fertility delta empirically before committing the $50K, and only commit if the saving compounds favorably across 50B pretraining tokens.
Real products, models, and research that use this idea.
1LegalBERT and Legal-RoBERTa (2020-2022 era) used domain-specific tokenizers tuned for case law, demonstrating the sequence-length wins on legal corpora.
2BloombergGPT (2023) trained a domain-specific 50K BPE for financial text, showing the same domain-tokenizer pattern at scale.
3Galactica (Meta, 2022) used a domain-specific tokenizer for scientific text with special tokens for citations, math, and chemical formulas.
4Code Llama and DeepSeek-Coder use code-aware BPE tokenizers; the legal-domain design parallels their approach but for a different specialty.
5HuggingFace tokenizers and tiktoken-style libraries both support byte-level BPE training on domain corpora with a few lines of code, making the engineering investment small.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the calculus change if the budget were $500K instead of $50K?
Sign in or go Pro to see the approach for this follow-up.
QWhy insert structural tokens via preprocessing rather than learning them from raw text via BPE merges?
Sign in or go Pro to see the approach for this follow-up.
QIf your validation experiment showed the domain BPE saves only 8 percent versus cl100k_base, what would you do?
Sign in or go Pro to see the approach for this follow-up.
QHow would you justify the 64K vocab choice if a stakeholder pushes for 128K because 'Llama 3 uses 128K'?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Choosing a vocab size based on what other models use (32K Llama 2, 128K Llama 3) without computing the embedding parameter cost as a fraction of the target model size.