Detailed answer

A token is an integer id between 0 and vocab_size minus 1, drawn from a fixed vocabulary defined by a specific tokenizer. The id points to a particular byte sequence. For byte-level BPE tokenizers (tiktoken's cl100k_base and o200k_base, the Llama 3+ tokenizer, newer Mistral, DeepSeek V4) the byte sequence is a chunk of UTF-8 bytes. It usually represents a sub-word fragment of text, but sometimes a whole short word or a piece of whitespace.

The id is the only thing the model ever sees. The embedding matrix is indexed by token id: each row is a learned vector for the token at that index. The output projection (lm_head) produces a logit per token id at every position. The model has no separate concept of characters; it works entirely in this discrete vocabulary.

The split between text and ids is strict. On the input side, the tokenizer converts the text string into a list of ids. On the output side, the tokenizer converts the model's emitted ids back into bytes and decodes them as UTF-8 to recover text. The model itself is blind to bytes.

Character-level tokenization gives every character its own token. The vocabulary is tiny: a few hundred entries for the Latin alphabet plus punctuation and digits, or 256 entries if you go byte-level. There is never an unknown input under byte-level character tokenization (and even pure character-level handles unknown characters more gracefully than word-level handles unknown words).

The failure mode is sequence length. A paragraph that fits in 300 subword tokens under o200k_base might require 1200 character tokens. Attention cost is O(n^2) in sequence length per layer, so 4x tokens means 16x attention compute per layer per forward pass. KV cache memory scales linearly with sequence length, so memory is 4x. For a fixed compute and memory budget, the effective context window shrinks by the same multiplier.

This is the main reason character-level lost. Attention's quadratic cost makes long sequences expensive, and character-level tokenization is the choice most aggressively at odds with attention's scaling behavior. Work like ByT5 (byte-level T5) has explored token-free models that operate directly on bytes. They remain niche because the compute cost is hard to justify against subword baselines.

A word-level tokenizer builds its dictionary once, from the training corpus. Every word it has seen gets an ID. Everything else is out of luck.

When a word-level model trained before 2022 encounters "ChatGPT", that string is not in the dictionary, so the tokenizer emits a single [UNK] token. The model receives a generic unknown marker with no information about the actual word. It cannot tell "ChatGPT" from "xqzptl". Both are just [UNK].

This is the out of vocabulary problem, and it is fatal for general language. New product names, proper nouns, technical terms, typos, and rare inflections appear constantly, and a closed vocabulary destroys all of them at the input. You cannot enlarge the dictionary to cover every possible string, because the space of possible words is effectively unbounded. So word-level is short and efficient but blind to anything new.

BPE builds its vocabulary by counting adjacent symbol pairs in a training corpus. It merges the most frequent pair, again and again, until it hits the target vocabulary size. The English words hello and world rank among the few thousand most common tokens in any web-scale English corpus. So BPE merges their character sequences into single vocabulary entries very early in training.

As a result, any modern OpenAI tokenizer (cl100k_base, o200k_base) maps each of these words to one integer. You can verify this in two lines of Python with the tiktoken library. The same logic applies to Anthropic, Google, and Meta tokenizers for these words because they all train on broadly similar English corpora.

Capitalization can matter. hello and Hello are usually separate tokens in the vocabulary. Both forms appear often enough in training data to earn their own entries. But they are still one token each.