Using tiktoken with cl100k_base, tokenize the following number strings:
- '1024'
- '2048'
- '100000'
Predict whether each is a single token or splits, and describe how inconsistent number tokenization causes LLMs to fail at multi-digit arithmetic.
In cl100k_base, the tokenizer used by recent OpenAI models, numbers are split into chunks of at most three digits. So `1024` becomes `102` plus `4`, `2048` becomes `204` plus `8`, and `100000` becomes `100` plus `000`. Even short-looking numbers split once they exceed three digits.
The reason this breaks arithmetic is that the model only ever sees these chunks, never the individual digits. To add two numbers correctly you line up ones, tens, and hundreds and carry between them. But the tokenizer splits numbers at three-digit boundaries, not along those place-value lines.
So when the model sees 1024 as the chunks 102 and 4, it has no clean access to the 1, the 0, the 2, and the 4 as separate place-valued quantities. It cannot run the digit by digit procedure you learned in school. Instead it falls back on patterns memorized from training, which works for famous sums and fails for arbitrary ones.
The trap is assuming the model adds digit by digit. It does not, because cl100k_base never hands it single digits for multi-digit numbers.
Here is the **prediction**. cl100k_base caps numeric runs at three digits, so `1024` is `['102', '4']`, `2048` is `['204', '8']`, and `100000` is `['100', '000']`. Every number longer than three digits splits, and the split point follows the three-digit grouping rule, not place value.
That mismatch is the whole problem. Correct addition needs a stable atomic unit, the single digit, so you can align ones with ones and propagate carries. The encoding cl100k gives the model multi-digit chunks instead. The chunk `102` from `1024` carries no addressable internal 1, 0, 2 the model can align.
The chunking is also misaligned with the operands you actually add. `1024` is `['102', '4']` while `1025` is `['102', '5']`. The model sees a shared `102` chunk and a varying final digit rather than four positioned digits. A pattern keyed to one chunk sequence does not generalize to a number whose chunking differs. The model ends up memorizing famous results rather than computing, which is why it nails 2 + 2 and stumbles on 471 + 829. The mitigation is to offload exact math to a code interpreter or to space out digits.
Quick reference
TL;DR
cl100k_base caps digit runs at three digits, so even 1024 splits into ['102','4']; that non place value chunking is why LLMs fail at exact arithmetic.
Imagine doing long addition, but instead of seeing each digit, you only get colored cards. The number 1024 is one blue card; the number 100000 is two green cards glued in a weird spot. You were taught to add by lining up the ones, tens, and hundreds, but the cards do not split along those lines. Worse, the number right next to 1024 might come as totally different cards, so a trick that worked for one number is useless for its neighbor. That is how a language model sees numbers. It rarely gets clean single-digit pieces, so it cannot line up place values, and it ends up guessing from patterns it memorized rather than actually calculating.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You need exact, auditable arithmetic on arbitrary multi-digit numbers. Token-space computation is unreliable, so route the calculation to a code interpreter rather than trusting the model.
Your numbers are domain literals the model has memorized, like common powers of two. The model may recall these correctly, but relying on memorization does not generalize and breaks on unfamiliar values.
You are estimating token cost from character counts for numeric-heavy text. Numbers chunk at the three-digit cap, so a number under four digits is one token while longer ones are several, and character-based estimates drift badly.
Quick scan · 2–4 min
Everything important, quickly.
4 min: three-digit chunk cap + no place-value primitive + carries crossing chunk boundaries + right to left grouping + tool-use mitigation.
Real-world examples
Real products, models, and research that use this idea.
1ChatGPT routes exact arithmetic to a Python code interpreter rather than computing multi-digit math in token space.
2OpenAI's o200k_base tokenizer caps numeric runs to one to three digit chunks, making number splitting more consistent than cl100k_base.
3Llama 4 models split digits individually in their numeric handling, a deliberate choice to give the model a stable place-value primitive.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you design a tokenizer change that materially improves multi-digit arithmetic?
Sign in or go Pro to see the approach for this follow-up.
QWhy does spacing digits like '4 7 1' sometimes help the model add correctly?
Sign in or go Pro to see the approach for this follow-up.
QIf a model gets famous sums right but random ones wrong, what does that tell you about its mechanism?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming the model adds digit by digit; cl100k hands it multi-digit chunks split at three-digit boundaries, not place-value boundaries, so it pattern-matches instead of computing.