Why do LLMs frequently fail at arithmetic involving multi-digit numbers? Trace the failure to tokenization.
Explain why modern LLMs are unreliable at basic arithmetic like 47 × 83 or 1024 + 999, tracing the root cause to how BPE tokenizes numbers. Include at least one concrete example of how inconsistent tokenization creates the failure.
LLMs fail at multi-digit math because cl100k chunks numbers at a three-digit cap, not at place-value boundaries, denying the model a stable single-digit unit to align columns and carry.
Think of learning to add with flashcards. Normally each card shows one digit, so you can stack the ones, the tens, the hundreds and carry between them. Now imagine someone gives you cards that glue up to three digits together, counting from the left. The glue lines rarely match the ones, tens, and hundreds. You can no longer line up the columns, so you stop calculating and just try to remember answers you have seen before. That is the situation a language model is in. The tokenizer hands it number chunks of up to three digits, not ones and tens, so it memorizes famous sums and guesses on the rest.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
It feels paradoxical that a model fluent enough to write working code can get 47 × 83 wrong. The resolution is that fluency and arithmetic draw on different requirements, and the tokenizer satisfies the first while quietly sabotaging the second. The model operates on tokens, and the way numbers become tokens has nothing to do with how arithmetic works.
The failure traces back to a specific source; it is not just that 'LLMs are bad at math'. The chain is concrete. cl100k chunks numbers at a three-digit cap, and that cap does not respect place value. Place value is exactly what carrying and column alignment need. Once you see that chain, the symptoms, including why errors seem random, fall into place.
Three-digit chunking, not place value
cl100k_base does not let numbers merge freely. Its pre-tokenization regex caps any run of digits at three. So no token ever spans more than three digits, no matter how common the full number is. This bounds how many distinct number tokens the vocabulary must carry.
Within that cap a run of one to three digits like 512 can be a single token, but anything longer is split first. So 1024 becomes ['102', '4'], 2048 becomes ['204', '8'], and 100000 becomes ['100', '000']. The leading three-digit group is taken, then the next.
Nothing in this process aligns to place value. The tokenizer has no notion of ones, tens, and hundreds. It cuts at the three-digit cap, which is often not where arithmetic would want the cut to be.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- ChatGPT offloads exact arithmetic to a Python code interpreter instead of computing multi-digit math in token space.
- OpenAI's o200k_base keeps the same one to three digit cap as cl100k_base, so numbers like 1024 and 2048 split the same way in both.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does spacing digits like '4 7 × 8 3' improve the model's arithmetic?
Spaces force each digit into its own token, restoring a per-digit place-value primitive the model can align and carry across.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming model size or training for arithmetic errors; the deeper cause is that cl100k's three-digit chunking never gives the model a stable single-digit token to compute place-value math with.
60 second bullets to scan on the way to the call.
Why cl100k_base splits numbers at a three-digit cap instead of by digit position
Which atomic unit reliable addition with carries needs
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.