Tiktoken

Your teammate mentions BPE at a code review. What are they talking about?

You need to count tokens locally. When do you reach for tiktoken versus SentencePiece?

A 1200-word system prompt, 200-word query, 150-word reply on GPT-5.5. Estimate the token cost, then name what doubles it.

You need the exact token count for a Claude Sonnet 4.6 prompt before sending it. Which method actually works?

You are about to ship a feature that calls GPT-5.5 per request. Walk through how you estimate the token cost before launch.

What would break if you tokenized a Llama prompt using tiktoken's cl100k_base instead of the Llama tokenizer?

Predict whether tokenizer.encode('world') and tokenizer.encode(' world') produce the same token IDs in tiktoken.

What does the '200k' in o200k_base signify, and what was improved compared to cl100k_base?