Explain why modern LLMs are unreliable at basic arithmetic like 47 × 83 or 1024 + 999, tracing the root cause to how BPE tokenizes numbers. Include at least one concrete example of how inconsistent tokenization creates the failure.
Free · 2 AI evals / day
LLMs are unreliable at multi-digit arithmetic because of how their tokenizer chops up numbers. cl100k_base caps any run of digits at three, so it splits numbers at three-digit boundaries, not by place position.
That matters because adding numbers correctly relies on seeing each digit and its place. You stack the ones, add, carry into the tens, and so on. The model never gets that clean view. A number like `1024` arrives as the chunks `102` and `4`, so the model has no separate handle on the 1, 0, and 2, and the final 4 sits in a different token.
Without access to individual digits, the model cannot run the carry and align procedure you learned in school, and a carry would have to cross from the `4` chunk into the `102` chunk. Instead it leans on sums it happened to memorize from training. That is why it gets `2 + 2` right but stumbles on `47 × 83`. A practical fix is to let the model call a calculator or code interpreter for exact math.
The root cause is tokenization. cl100k_base caps digit runs at three digits, so it splits numbers at three-digit boundaries, **not** at place-value lines. The model rarely gets the one thing reliable arithmetic needs: a stable single-digit unit.
Numbers of three digits or fewer like 512 can be a single token, but anything longer splits: `1024` is `['102', '4']`, `2048` is `['204', '8']`. The boundaries land at the three-digit cap, never at the thousands or hundreds line.
Concrete example: 1024 + 999. In cl100k_base `1024` is `['102', '4']` and 999 is one token. So the model must produce 2023 without ever accessing the individual digits as separate quantities. A carry out of the ones place must cross from the `4` chunk into the `102` chunk. Now compare 1025 + 998: `1025` is `['102', '5']`, so the carry crosses the boundary differently and whatever pattern the model learned for the first case does not transfer.
Multiplication compounds this. Digit by digit multiplication needs every place value, but if `47` is one token the model has no internal 4 and 7 to work with. It falls back on recalling memorized results, which is not a procedure.
The practical mitigation is to route exact math to a code interpreter, or to space out digits like `4 7` so each becomes its own token.
Quick reference
TL;DR
LLMs fail at multi-digit math because cl100k chunks numbers at a three-digit cap, not at place-value boundaries, denying the model a stable single-digit unit to align columns and carry.
Think of learning to add with flashcards. Normally each card shows one digit, so you can stack the ones, the tens, the hundreds and carry between them. Now imagine someone gives you cards that sometimes glue several digits together in random spots, and the gluing changes from one number to the next. You can no longer line up the columns, so you stop calculating and just try to remember answers you have seen before. That is the situation a language model is in. The tokenizer hands it number chunks cut by how common they were in text, not by ones and tens, so it memorizes famous sums and guesses on the rest.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You need exact arithmetic that must be correct and auditable. Token-space computation is unreliable, so route the calculation to a code interpreter rather than the model itself.
You are tempted to trust the model on a famous constant or power of two. Correct recall of memorized values does not generalize, so the same approach breaks on arbitrary numbers.
You want a quick estimate where small errors are acceptable. Approximate reasoning is fine, but do not present token-space output as exact when precision matters.
Quick scan · 2–4 min
Everything important, quickly.
3 min: frequency-based number splits + missing digit primitive + neighbor instability + multiplication compounding + tool-use and digit-spacing fixes.
Real-world examples
Real products, models, and research that use this idea.
1ChatGPT offloads exact arithmetic to a Python code interpreter instead of computing multi-digit math in token space.
2OpenAI's o200k_base caps numeric runs to one-to-three-digit chunks, making number tokenization more consistent than cl100k_base.
3Prompt-engineering guides recommend space-separated digits like '4 7' to force per-digit tokens and improve arithmetic accuracy.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does spacing digits like '4 7 × 8 3' improve the model's arithmetic?
Sign in or go Pro to see the approach for this follow-up.
QIf a model nails common sums but fails arbitrary ones, what does that reveal about its mechanism?
Sign in or go Pro to see the approach for this follow-up.
QWould a character-level model solve the arithmetic problem outright?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Blaming model size or training for arithmetic errors; the deeper cause is that cl100k's three-digit chunking never gives the model a stable single-digit token to compute place-value math with.