Vocabulary size should match how many meaningful patterns a domain can actually produce. For natural language, a rich alphabet and millions of word forms justify 32k to 200k tokens. DNA is the opposite case.
DNA uses exactly four characters: A, C, G, and T. The meaningful patterns are **k-mers**, short subsequences of a fixed length. The number of distinct k-mers grows as four to the power k:
$$\text{number of k-mers} = 4^{k}$$
That gives 64 codons at k=3, 1,024 at k=5, and 4,096 at k=6. Past length six, k-mers get rare enough that a tokenizer trained on real DNA rarely forms merges for them.
So a vocabulary of 4,096 to 8,192 captures every useful k-mer up to length six, plus room for a few longer biological motifs and special tokens. A 100k vocabulary would leave most slots empty forever, and the 256-token byte-level option is too small to capture k-mer structure at all. The right size mirrors the domain's real pattern count.
DNA has a four-character alphabet, so the meaningful patterns are **k-mers** and their count is `4^k`. That gives 4,096 unique 6-mers. That is the practical ceiling for patterns common enough to earn a merge during training.
A vocabulary of 4,096 to 8,192 covers all k-mers up to length six. It leaves headroom for frequent longer motifs like restriction sites and promoters, and it includes special tokens, all at close to full utilization.
The distractors fail on efficiency grounds. A 50k-100k vocabulary borrowed from NLP wastes the great majority of its slots on sequences that never appear. Every unused row is dead embedding memory that receives no gradient. The 256-token byte-level option swings too far the other way: it captures single nucleotides but loses the k-mer context, like codons and binding sites, that carries the biological signal. A million tokens is simply incoherent for a four-letter alphabet.
Quick reference
TL;DR
DNA's four-letter alphabet yields at most 4^6 = 4,096 useful 6-mers, so a 4k-8k vocabulary covers the meaningful patterns without wasting embedding rows.
Imagine a language that only ever uses four letters: A, C, G, and T. If you try to build a dictionary of useful 'words' from those letters, the longest ones that carry real meaning are about six letters long, and there are only 4 × 4 × 4 × 4 × 4 × 4 = 4,096 of those.
Now imagine ordering a 100,000-word dictionary for a language that can only ever make a few thousand words. Almost every page would stay blank forever. Picking a 4,000 to 8,000 word dictionary instead fits the language exactly: enough room for every useful pattern, and almost no wasted pages.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Sizing a vocabulary for a rich natural language with broad morphology The productive token space there is millions of word forms, so the four to the k ceiling does not apply and 32k-200k is appropriate.
A task that needs single-nucleotide resolution, such as base-level variant calling Merging nucleotides into k-mers blurs the exact positions the task must read, so a character-level vocabulary serves better.
Initializing a genomics model from a pretrained NLP checkpoint that requires the original 50k vocabulary Shrinking the vocabulary breaks the inherited embedding and head, so the architecture constraint can override the efficiency argument.
Quick scan · 2–4 min
Everything important, quickly.
2 min: four-letter alphabet + 4^k k-mer count + 6-mer ceiling at 4,096 + 4k-8k target + wasted rows at 100k + protein analogy.
Real-world examples
Real products, models, and research that use this idea.
1DNABERT-2 trains a BPE tokenizer on multi-species genomes with a roughly 4,096-token vocabulary, capturing variable-length k-mers instead of a fixed k.
2InstaDeep's Nucleotide Transformer enumerates all 4^6 = 4,096 hexamers directly as a fixed-k vocabulary, guaranteeing every valid 6-mer is present.
3Meta's ESM protein models tokenize with a 33-token vocabulary of 20 amino acids plus special tokens, since a 20-letter alphabet needs no BPE at all.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow should pre-tokenization be configured for DNA, and why does the NLP whitespace default break it?
Sign in or go Pro to see the approach for this follow-up.
QFor a 20-amino-acid protein model, what vocabulary size and algorithm would you choose, and how does it differ from DNA?
Sign in or go Pro to see the approach for this follow-up.
QHow would you empirically confirm your chosen vocabulary size is well utilized after training?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Copying NLP vocabulary sizes of 50k-100k onto a four-letter alphabet, which leaves over 90% of the embedding table permanently empty and untrained.