You are building a domain specific LLM for genomics. Your training data consists of DNA sequences (e.g., 'ACGTTTACGCGATCG...'). Describe how you would train a custom tokenizer, including: the algorithm choice, vocabulary size decision, pre-tokenization configuration, and the HuggingFace tokenizers API calls you would use.
Train BPE on a genomics-only corpus with pre-tokenization disabled (DNA has no spaces), set vocab_size to 4k-8k to match the 4^6 k-mer ceiling, and add domain special tokens.
Imagine teaching a reading machine to read DNA, which is one endless string of four letters with no spaces. A normal reading machine looks for spaces to find where words end, but here there are none, so you must tell it to ignore spaces and just hunt for repeated chunks. Then you tell it how big its notebook of chunks should be. Since there are only four letters, the number of useful chunks is small, a few thousand, so a notebook of four to eight thousand entries is plenty. Finally, you train it only on real DNA, never on English, so the chunks it memorizes are the ones that actually show up in genomes.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Building a tokenizer for a genomics LLM forces you to question every default that an NLP tokenizer bakes in. DNA has no spaces, a four-letter alphabet, and meaningful structure at fixed k-mer lengths rather than at morpheme boundaries. Each of those facts overturns a standard assumption, and missing one quietly cripples the model.
The interview is checking whether you reason from the domain rather than copy a recipe. The strongest answers name the pre-tokenization trap first, because it is the failure that produces a tokenizer that trains without error yet learns nothing useful. The second strongest move is to derive the vocabulary size from the k-mer count rather than reaching for a familiar 50k.
A full answer also touches algorithm choice, special tokens, and corpus construction, because a production tokenizer is more than a vocabulary size. We will walk through the pre-tokenization problem, derive the vocabulary size from the k-mer space, discuss corpus and special-token construction, cover when Unigram beats BPE, and show how a single restriction site demonstrates the whole pipeline working.
The pre-tokenization trap
Every standard HuggingFace pipeline runs a pre-tokenization step before BPE learns merges. The common pre-tokenizers split on whitespace, on whitespace and punctuation, or on a regex pattern, all of which assume text has visible word boundaries.
DNA has none. A sequence like 'ACGTTTACGCGATCGGAATTC' can run for millions of characters with no delimiter. If whitespace splitting is active, the entire sequence becomes a single word, and BPE can only merge characters inside that one unit, which collapses to learning almost nothing.
The fix is one line: set the pre-tokenizer to None so the raw character stream flows directly into BPE. Now merges can form anywhere a k-mer recurs, which is exactly the behavior the domain needs. This is the highest-leverage decision in the whole pipeline, and it is invisible if you only check that training completed.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
# [UNK] absorbs ambiguous bases (N, Y, R) under character-level BPE
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
# CRITICAL: DNA has no word boundaries, so disable pre-tokenization.
# Leaving the default on makes the trainer treat a chromosome as one word.
tokenizer.pre_tokenizer = None
# 4^6 = 4,096 hexamers is the productive ceiling; 8192 leaves headroom.
trainer = BpeTrainer(
vocab_size=8192,
special_tokens=["[UNK]", "[PAD]", "[MASK]", "[BOS]", "[EOS]",
"[STRAND+]", "[STRAND-]"],
min_frequency=2, # drop single-occurrence k-mers (often read errors)
show_progress=True,
)
# Train on genomics corpora ONLY, never general text.
tokenizer.train(files=["hg38.txt", "bacterial.txt", "viral.txt"], trainer=trainer)
tokenizer.save("dna_tokenizer.json")
# Sanity check: GAATTC is the EcoRI restriction site, very common in hg38.
out = tokenizer.encode("ATCGTTACGCGATCGGAATTCATCG")
print(out.tokens) # expect GAATTC to surface as a single token
print(len(tokenizer.get_vocab()))Real products, models, and research that use this idea.
- DNABERT-2 swapped DNABERT's fixed 6-mer scheme for BPE trained on multi-species genomes at a roughly 4,096-token vocabulary, learning variable-length motifs from data.
- InstaDeep's Nucleotide Transformer instead enumerates all 4^6 = 4,096 hexamers as a fixed vocabulary, guaranteeing every valid 6-mer is present without training merges.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle reverse-complement strands, and should they map to the same token IDs as the forward strand?
Weigh explicit strand-marker tokens against training on both orientations; discuss how each affects the model's ability to generalize across strands.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Leaving whitespace pre-tokenization on for DNA, which treats an entire chromosome as one word so the trainer can only learn merges inside that single giant unit.
60 second bullets to scan on the way to the call.
Algorithm choice between BPE and Unigram for DNA
Why pre-tokenization must be disabled for contiguous sequences
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.