Pick the true consequence of swapping tokenizers after 1T tokens

A tokenizer swap is not a configuration change, it is a structural model migration. The transformer body, the attention and feedforward layers in the middle, does operate on continuous vectors and is agnostic to which token IDs feed it. But the input embedding layer and the output projection are lookup tables indexed by token ID. Those layers are where the swap breaks. Under the new tokenizer, the same text maps to different IDs. Many IDs that used to mean one thing now mean something completely different, and many new IDs exist that the embedding table has no row for. A naive swap feeds the model scrambled inputs and asks it to predict from a head that no longer matches the vocabulary. Real tokenizer migrations involve embedding surgery: copy rows for tokens that exist in both vocabularies, initialize new rows from subword compositions or random values, and run an adaptation phase with a re-warmed learning rate to let the model relearn the new mapping. Even after that, loss per token is measured against a different vocabulary size, so pre-swap and post-swap loss curves are not comparable; a larger vocab mechanically shifts the per-token loss scale.

**The transformer body is token-agnostic; the embedding and output layers are not.** The input embedding matrix is shape `(V, d_model)`, indexed by token ID. The output projection (often tied to the input embedding) is shape `(d_model, V)`, producing logits per token ID. Both layers are coupled to the specific vocabulary the model was trained against. A tokenizer swap changes the meaning of every ID. The same text under the new tokenizer produces a different ID sequence; many old IDs are remapped to new strings or no longer exist; many new IDs are introduced that the old embedding has no row for. Feeding the new ID sequence into the old embedding lookup table is a category error: the model sees random vectors where it expects learned representations. The correct migration has three stages. First, embedding surgery: copy rows for tokens that exist in both vocabularies, initialize new tokens by composing the embeddings of their subword pieces under the old tokenizer or by random initialization, and re-initialize the output head correspondingly. Second, a re-warmed adaptation phase where the learning rate ramps back up briefly so the freshly-touched layers can catch up. Third, accept that loss curves are not comparable across the swap: a larger vocabulary mechanically lowers per-token cross-entropy because each token carries more information.