Token 1234 means 'the' in one model and garbage in another. Why?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Because each tokenizer builds its own vocabulary independently, and the integer 1234 is just a row index into that tokenizer's specific vocabulary table. cl100k_base assigned id 1234 to one subword during its BPE training; the Llama 3 tokenizer assigned id 1234 to a completely different subword during its own training. The number is a coordinate inside one vocabulary, not a global identifier. Feeding cl100k_base ids into a Llama model means every embedding lookup is wrong, producing garbage. The only portable representation between models is the raw text.

Your answer

Token 1234 means 'the' in one model and garbage in another because each tokenizer builds its own vocabulary independently. **Token ids** are just row numbers inside that specific vocabulary table. When a tokenizer is trained (via BPE merges or Unigram probabilities), it assigns an integer id to every entry. cl100k_base might assign id 1234 to the subword 'the'. The Llama 3 tokenizer, trained on different data with a different merge history, might assign id 1234 to a fragment like 'ruct'. The integer carries no global meaning. The model's weights are coupled to these coordinates. The embedding matrix has one row per vocabulary entry, so row 1234 in a GPT-4 model holds the learned representation for whatever cl100k_base put at position 1234. Row 1234 in a Llama 3 model holds the representation for whatever the Llama 3 tokenizer assigned there. Feeding cl100k_base ids into a Llama 3 model means every row lookup is wrong, which is why the output looks like garbage. The practical rule: token ids never cross model boundaries. The only portable representation is raw text. Detokenize on one side, re-tokenize on the other, and accept that the resulting ids and sequence lengths will differ.

**A token id** is the integer index of a token in a tokenizer's vocabulary, and it serves two specific roles inside the model. At input time, the id indexes the embedding matrix (shaped vocab_size by d_model). Token id 1234 selects the 1234th row, producing a d_model-dimensional vector that becomes the input representation for that position. Every subsequent layer operates on these vectors, not on the ids themselves. At output time, the model produces a vector of vocab_size logits per position. The id of the sampled entry is the model's prediction. The decoder maps that id back to a string via the vocabulary table, and the strings concatenate into generated text. The id is meaningful only relative to the tokenizer that produced it. id 1234 in tiktoken cl100k_base, in o200k_base, in the Llama 3 tokenizer, and in Gemma's SentencePiece are four different subwords with no relationship. The integer is a coordinate inside one vocabulary, not a global name. This is why every model is tightly coupled to its tokenizer. The model weights were trained against specific embedding rows for specific tokens; swapping tokenizers feeds the model misindexed embeddings. The only portable representation between models is raw text. Decode on one side, re-encode on the other, and accept that the token sequences will differ in length, ids, and splits.

Quick reference

TL;DR

Each tokenizer builds its own vocabulary independently, so id 1234 is a row index into one specific table. Different tokenizer, different table, different meaning. Raw text is the only portable representation.

Memory aid

Easy to grasp

Imagine each tokenizer has a numbered locker room. Token id 47 means 'locker 47 in this specific room'. Inside locker 47 is the embedding vector representing one piece of text. Now imagine a different building with its own locker room. Locker 47 there holds something completely different. There is no relationship between locker 47 in building A and locker 47 in building B. Token ids work the same way. The id is a coordinate inside one tokenizer's vocabulary, not a global name. The only way two models can share information is through the raw text: detokenize on one side, re-tokenize on the other. Passing token ids directly between models is like mailing someone your gym locker number and expecting it to mean something at their gym.

Key concepts

Token ID Embedding Matrix Vocabulary Size Tokenizer-Model Coupling

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: define a token id as an integer index into the tokenizer's vocabulary, name the two model components it touches (embedding matrix at input, lm_head at output), explain non-portability with a concrete example, and close with the rule that raw text is the only safe representation across model boundaries.

Real-world examples

Real products, models, and research that use this idea.

In tiktoken cl100k_base the token ' the' (with leading space) has a specific id; that same surface form in o200k_base has a different id because the vocabularies differ.
Llama 3's tokenizer assigns specific ids to chat template control tokens; mismatching these breaks apply_chat_template and generation behavior.
When fine-tuning a model and adding domain-specific tokens, you append new rows to the embedding matrix and lm_head; those ids become useful only after training exercises them.
Cross-model evaluation pipelines (comparing GPT-5.5 and Claude Opus 4.7 on the same prompt) always pass through text space at the boundary, not token id space.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat happens inside the model the moment a token id is read from the input?

QWhy is adding new tokens to a pretrained model such a heavy operation?

QHow can two tokenizers produce similar looking splits for common inputs but wildly different ids?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating token ids as portable between models. They are integer indices into a specific tokenizer's vocabulary; the same id in two tokenizers points to two unrelated entries.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.