Why do LLMs frequently fail at arithmetic involving multi-digit numbers? Trace the failure to tokenization.
Explain why modern LLMs are unreliable at basic arithmetic like 47 × 83 or 1024 + 999, tracing the root cause to how BPE tokenizes numbers. Include at least one concrete example of how inconsistent tokenization creates the failure.
LLMs fail at multi-digit math because cl100k chunks numbers at a three-digit cap, not at place-value boundaries, denying the model a stable single-digit unit to align columns and carry.
Think of learning to add with flashcards. Normally each card shows one digit, so you can stack the ones, the tens, the hundreds and carry between them. Now imagine someone gives you cards that sometimes glue several digits together in random spots, and the gluing changes from one number to the next. You can no longer line up the columns, so you stop calculating and just try to remember answers you have seen before. That is the situation a language model is in. The tokenizer hands it number chunks cut by how common they were in text, not by ones and tens, so it memorizes famous sums and guesses on the rest.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
It feels paradoxical that a model fluent enough to write working code can botch 47 × 83. The resolution is that fluency and arithmetic draw on different requirements, and the tokenizer satisfies the first while quietly sabotaging the second. The model operates on tokens, and the way numbers become tokens has nothing to do with how arithmetic works.
This question asks you to trace the failure back to its source rather than wave at 'LLMs are bad at math'. The chain is concrete: cl100k chunks numbers at a three-digit cap, that cap does not respect place value, and place value is exactly what carrying and column alignment need. Once you see that chain, the symptoms, including why errors seem random, fall into place.
We will build the argument from the tokenizer up, ground it in a worked example, and end with the mitigations that actually hold in production.
Three-digit chunking, not place value
cl100k_base does not let numbers merge freely. Its pre-tokenization regex caps any run of digits at three, so no token ever spans more than three digits, no matter how common the full number is. This bounds how many distinct number tokens the vocabulary must carry.
Within that cap a one-to-three-digit run like 512 can be a single token, but anything longer is split first. So 1024 becomes ['102', '4'], 2048 becomes ['204', '8'], and 100000 becomes ['100', '000']. The leading three-digit group is taken, then the next.
The takeaway is that nothing in this process aligns to place value. The tokenizer has no notion of ones, tens, and hundreds; it cuts at the three-digit cap, which is almost never where arithmetic would want the cut to be.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- ChatGPT offloads exact arithmetic to a Python code interpreter instead of computing multi-digit math in token space.
- OpenAI's o200k_base caps numeric runs to one-to-three-digit chunks, making number tokenization more consistent than cl100k_base.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does spacing digits like '4 7 × 8 3' improve the model's arithmetic?
Spaces force each digit into its own token, restoring a per-digit place-value primitive the model can align and carry across.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming model size or training for arithmetic errors; the deeper cause is that cl100k's three-digit chunking never gives the model a stable single-digit token to compute place-value math with.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.