For a genomics tokenizer with a 4-character DNA alphabet, what vocabulary size range is likely sufficient?

Vocabulary size should match how many meaningful patterns a domain can actually produce. For natural language, a rich alphabet and millions of word forms justify 32k to 200k tokens. DNA is the opposite case. DNA uses exactly four characters: A, C, G, and T. The meaningful patterns are **k-mers**, short subsequences of a fixed length. The number of distinct k-mers grows as four to the power k: $$\text{number of k-mers} = 4^{k}$$ That gives 64 codons at k=3, 1,024 at k=5, and 4,096 at k=6. Past length six, k-mers get rare enough that a tokenizer trained on real DNA rarely forms merges for them. So a vocabulary of 4,096 to 8,192 captures every useful k-mer up to length six, plus room for a few longer biological motifs and special tokens. A 100k vocabulary would leave most slots empty forever, and the 256-token byte-level option is too small to capture k-mer structure at all. The right size mirrors the domain's real pattern count.

DNA has a four-character alphabet, so the meaningful patterns are **k-mers** and their count is `4^k`. That gives 4,096 unique 6-mers. That is the practical ceiling for patterns common enough to earn a merge during training. A vocabulary of 4,096 to 8,192 covers all k-mers up to length six. It leaves headroom for frequent longer motifs like restriction sites and promoters, and it includes special tokens, all at close to full utilization. The distractors fail on efficiency grounds. A 50k-100k vocabulary borrowed from NLP wastes the great majority of its slots on sequences that never appear. Every unused row is dead embedding memory that receives no gradient. The 256-token byte-level option swings too far the other way: it captures single nucleotides but loses the k-mer context, like codons and binding sites, that carries the biological signal. A million tokens is simply incoherent for a four-letter alphabet.