Real world examples

Real products, models, and research that use this idea.

• OpenAI's tiktoken library exposes cl100k_base (GPT-4, GPT-3.5-turbo, text-embedding-ada-002) and o200k_base (GPT-4o, o1, o3, GPT-5), letting you count tokens locally before sending a request.
• Anthropic's Claude tokenizer is proprietary; the only reliable way to count tokens for Opus 4.7 or Sonnet 4.6 is the messages.count_tokens endpoint.

Real products, models, and research that use this idea.

• GPT-4o with o200k_base tokenizes about four English characters per token on average, but Chinese text needs about two thirds of a token per character (cl100k_base often more than one), reflecting the Latin-script bias.
• Old character-level models like ByT5 (byte-level T5) exist but are niche; they trade compute for the ability to handle arbitrary bytes without subword vocabulary mismatches.

Real products, models, and research that use this idea.

• OpenAI's GPT-4o and GPT-5 use the byte-level BPE tokenizer o200k_base, which keeps English near 1.3 tokens per word while never emitting an unknown token.
• Meta's Llama 3 and Llama 4 use tiktoken-style BPE vocabularies (about 128K and 200K entries) so they can represent code, proper nouns, and many languages without an OOV marker.

Real products, models, and research that use this idea.

• OpenAI bills its GPT-5 and o-series APIs in tokens, and tiktoken counts them locally; underestimating token counts is a common cost-projection bug.
• Anthropic's Claude models use their own tokenizer; the same string can have different counts across providers.

Real products, models, and research that use this idea.

• SentencePiece models trained with the default nmt_nfkc rule fold half-width katakana and ligatures before segmentation. Hugging Face's BertNormalizer leaves them unchanged and only lowercases and strips accents.
• Multilingual search pipelines normalize queries with NFKC so that a half-width and full-width form of the same Japanese term hit the same index entries.

Real products, models, and research that use this idea.

• OpenAI's tiktoken (cl100k_base, o200k_base) applies almost no normalization and treats input as raw bytes.
• Hugging Face tokenizer.json files for BERT family models include explicit normalizer configurations. bert-base-uncased lowercases and strips accents (NFD, then drops combining marks). bert-base-cased only cleans control characters and spaces out Chinese characters, with no Unicode normalization.

Real products, models, and research that use this idea.

• OpenAI's tiktoken source publishes the exact regex for cl100k_base and o200k_base. You can inspect them in the encoding registry and even compile and apply them yourself in Python.
• The cl100k_base regex specifically handles English contractions ('s, 't, 're, 've, 'm, 'll, 'd) as their own chunks, which is why 'don't' tokenizes consistently across casing variations.

Real products, models, and research that use this idea.

• OpenAI's tiktoken cl100k_base has 100,277 ids, special tokens included (used by GPT-4 and GPT-3.5-turbo); o200k_base has 200,019 (used by GPT-4o, o1, o3, GPT-5).
• Meta's Llama 3 uses a 128K-token byte-level BPE vocabulary, expanded from Llama 2's 32K SentencePiece vocabulary primarily to improve multilingual and code performance. Llama 4 grew to about 200K.

Real products, models, and research that use this idea.

• In tiktoken cl100k_base the token ' the' (with leading space) has a specific id; that same surface form in o200k_base has a different id because the vocabularies differ.
• Llama 3's tokenizer assigns specific ids to chat template control tokens; mismatching these breaks apply_chat_template and generation behavior.

Real products, models, and research that use this idea.

• tiktoken cl100k_base ships a 100K vocabulary for GPT-4 era models; o200k_base ships 200K for GPT-4o, o1, o3, and GPT-5.
• Llama 3 uses a 128K BPE vocabulary, quadrupled from Llama 2's 32K SentencePiece; Llama 4 grew to about 200K.

Real products, models, and research that use this idea.

• Llama 3.3 ships with a tokenizer_config.json that includes a Jinja2 chat_template defining the '<|start_header_id|>...<|end_header_id|>' format.
• Gemma 3 ships with a tokenizer_config.json declaring the '<start_of_turn>user...<end_of_turn>' chat template and turn marker ids.

Real products, models, and research that use this idea.

• Anthropic offers messages.count_tokens as the supported method for Claude 3 and the Claude 4 family, accepting the same payload as messages.create and returning input_tokens.
• OpenAI's tiktoken library ships encoding_for_model() that returns cl100k_base for GPT-4 and o200k_base for GPT-4o, GPT-5, and o-series models.

Real products, models, and research that use this idea.

• OpenAI's cl100k_base reserves id 100257 for <|endoftext|>, and a ChatML extension of it adds <|im_start|> at 100264; the chat API inserts its message markers itself.
• Llama 3 uses <|begin_of_text|>, <|start_header_id|>, <|end_header_id|>, and <|eot_id|> as chat template tokens (Llama 4 renames them <|header_start|>, <|header_end|>, and <|eot|>); exact formatting matters for quality.

Real products, models, and research that use this idea.

• Llama 4 Maverick served locally needs the tokenizer's add_bos_token config set correctly, since base-model quality drops when BOS is missing from the prompt.
• OpenAI's ChatML format, documented for gpt-3.5-turbo, fences each role with <|im_start|>system and <|im_end|>, and malformed markers break the role contract.