Token 1234 means 'the' in one model and garbage in another. Why?

Flashcard·Easy·4.0 · 0·~30s·Asked atLangChain Meesho·Relevant atOpenAI

Attempt it

Because each tokenizer builds its own vocabulary independently, and the integer 1234 is just a row index into that tokenizer's specific vocabulary table. cl100k_base assigned id 1234 to one subword during its BPE training; the Llama 3 tokenizer assigned id 1234 to a completely different subword during its own training. The number is a coordinate inside one vocabulary, not a global identifier. Feeding cl100k_base ids into a Llama model means every embedding lookup is wrong, producing garbage. The only portable representation between models is the raw text.

Your answer

Token 1234 means 'the' in one model and garbage in another because each tokenizer builds its own vocabulary independently. **Token ids** are just row numbers inside that specific vocabulary table. When a tokenizer is trained (via BPE merges or Unigram probabilities), it assigns an integer id to every entry. cl100k_base might assign id 1234 to the subword 'the'. The Llama 3 tokenizer, trained on different data with a different merge history, might assign id 1234 to a fragment like 'ruct'. The integer carries no global meaning. The model's weights are coupled to these coordinates. The embedding matrix has one row per vocabulary entry, so row 1234 in a GPT-4 model holds the learned representation for whatever cl100k_base put at position 1234. Row 1234 in a Llama 3 model holds the representation for whatever the Llama 3 tokenizer assigned there. Feeding cl100k_base ids into a Llama 3 model means every row lookup is wrong, which is why the output looks like garbage. The practical rule: token ids never cross model boundaries. The only portable representation is raw text. Detokenize on one side, re-tokenize on the other, and accept that the resulting ids and sequence lengths will differ.

**A token id** is the integer index of a token in a tokenizer's vocabulary, and it serves two specific roles inside the model. At input time, the id indexes the embedding matrix (shaped vocab_size by d_model). Token id 1234 selects the 1234th row, producing a d_model-dimensional vector that becomes the input representation for that position. Every subsequent layer operates on these vectors, not on the ids themselves. At output time, the model produces a vector of vocab_size logits per position. The id of the sampled entry is the model's prediction. The decoder maps that id back to a string via the vocabulary table, and the strings concatenate into generated text. The id is meaningful only relative to the tokenizer that produced it. id 1234 in tiktoken cl100k_base, in o200k_base, in the Llama 3 tokenizer, and in Gemma's SentencePiece are four different subwords with no relationship. The integer is a coordinate inside one vocabulary, not a global name. This is why every model is tightly coupled to its tokenizer. The model weights were trained against specific embedding rows for specific tokens; swapping tokenizers feeds the model misindexed embeddings. The only portable representation between models is raw text. Decode on one side, re-encode on the other, and accept that the token sequences will differ in length, ids, and splits.

TL;DR

Each tokenizer builds its own vocabulary independently, so id 1234 is a row index into one specific table. Different tokenizer, different table, different meaning. Raw text is the only portable representation.

Memory aid

Easy to grasp

Imagine each tokenizer has a numbered locker room. Token id 47 means 'locker 47 in this specific room'. Inside locker 47 is the embedding vector representing one piece of text. Now imagine a different building with its own locker room. Locker 47 there holds something completely different. There is no relationship between locker 47 in building A and locker 47 in building B. Token ids work the same way. The id is a coordinate inside one tokenizer's vocabulary, not a global name. The only way two models can share information is through the raw text: detokenize on one side, re-tokenize on the other. Passing token ids directly between models is like mailing someone your gym locker number and expecting it to mean something at their gym.

Key concepts

Token ID Embedding Matrix Vocabulary Size Tokenizer-Model Coupling

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A token id is one of the smallest objects in an LLM stack, but understanding it precisely clears up an entire category of cross-model bugs. The id is an integer index into a specific tokenizer's vocabulary. It has no meaning outside that tokenizer. The model uses it to index the embedding matrix at input and to label logit positions at output.

This deep dive walks through the mechanics: how ids interact with the model, why they are coupled to the tokenizer, the role of special tokens, and the implications for any pipeline that spans more than one model.

What a token id does inside the model

The token id has two specific jobs, one at each end of the transformer pipeline.

At input time, the id indexes the embedding matrix. The embedding matrix is a vocab_size by d_model array of trained parameters. Token id 1234 means 'use row 1234 as the input vector for this position'. That row is a d_model-dimensional vector that becomes the input to the first transformer block, typically after modification by a positional scheme like RoPE.

From that point the model does not use the id directly. Every transformer layer operates on continuous d_model-dimensional vectors. The discrete identity of the token is encoded in the embedding vector and the model's learned associations.

At output time, the final layer produces logits via a matrix multiply against the lm_head (shaped d_model by vocab_size). The result is a logit for every vocabulary position. Sampling picks one position, and that position is the token id of the predicted next token. The decoder maps that id back to the corresponding vocabulary entry and appends it to the output.

This is the entire path. Input ids become embedding vectors, transformer layers process them, lm_head produces logits, sampling picks an id, decoder maps it back to text. The id is the boundary between discrete text-space and continuous vector-space at both ends.

Why ids are not portable across tokenizers

Why model weights are coupled to the tokenizer

Special token ids

The cross-model interoperability rule

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

In tiktoken cl100k_base the token ' the' (with leading space) has a specific id; that same surface form in o200k_base has a different id because the vocabularies differ.
Llama 3's tokenizer assigns specific ids to chat template control tokens; mismatching these breaks apply_chat_template and generation behavior.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat happens inside the model the moment a token id is read from the input?

The id indexes the embedding matrix to produce a d_model-dimensional vector. That vector is modified by the positional encoding (or RoPE) and becomes the input to the first transformer block. From that point the model operates on continuous vectors; the discrete id is not used again until the final logit projection.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating token ids as portable between models. They are integer indices into a specific tokenizer's vocabulary; the same id in two tokenizers points to two unrelated entries.

60 second bullets to scan on the way to the call.

Define a token id as an integer index into a specific tokenizer's vocabulary.
Name the two model components token ids interact with (embedding matrix at input, lm_head at output).

Primary sources. Browse if you want the original framing.