What is the true consequence of swapping tokenizers after 1T tokens?

A tokenizer swap is not a configuration change. It is a structural model migration. The transformer body (the attention and feed-forward layers in the middle) works on continuous vectors. It does not care which token IDs (numbers for text pieces) feed it. But the input embedding layer and the output projection are lookup tables indexed by token ID. Those layers are where the swap breaks. Under the new tokenizer, the same text maps to different IDs. Many IDs that used to mean one thing now mean something different. Many new IDs also exist that the embedding table has no row for. A naive swap feeds the model scrambled inputs and asks it to predict from a head that no longer matches the vocabulary. Real migrations need embedding surgery. Copy rows for tokens that exist in both vocabularies, and initialize new rows from subword compositions or random values. Then run an adaptation phase with a re-warmed learning rate to let the model relearn the new mapping. Even after that, loss per token is measured against a different vocabulary size, so pre-swap and post-swap loss curves are not comparable. A larger vocab mechanically shifts the per token loss scale.

**The transformer body is token-agnostic; the embedding and output layers are not.** The input embedding matrix has shape `(V, d_model)` and is indexed by token ID. The output projection (often tied to the input embedding) has shape `(d_model, V)` and produces logits per token ID. Both layers are coupled to the vocabulary the model was trained against. A tokenizer swap changes the meaning of every ID. The same text under the new tokenizer produces a different ID sequence. Many old IDs are remapped to new strings or no longer exist. Many new IDs are introduced that the old embedding has no row for. Feeding the new ID sequence into the old embedding lookup table is a category error: the model sees learned vectors for the wrong tokens. The correct migration has three stages. First, do embedding surgery. Copy rows for tokens that exist in both vocabularies. Initialize new tokens by composing the embeddings of their subword pieces under the old tokenizer, or by random initialization. Re-initialize the output head the same way. Second, run a re-warmed adaptation phase. The learning rate ramps back up briefly so the freshly touched layers can catch up. Third, accept that loss curves are not comparable across the swap: a larger vocabulary mechanically raises per token cross-entropy because each token carries more information.