Why do LLMs frequently fail at arithmetic involving multi-digit numbers? Trace the failure to tokenization.

LLMs are unreliable at multi-digit arithmetic because of how their tokenizer chops up numbers. cl100k_base caps any run of digits at three, so it splits numbers at three-digit boundaries, not by place position. That matters because adding numbers correctly relies on seeing each digit and its place. You stack the ones, add, carry into the tens, and so on. The model never gets that clean view. A number like `1024` arrives as the chunks `102` and `4`, so the model has no separate handle on the 1, 0, and 2, and the final 4 sits in a different token. Without access to individual digits, the model cannot run the carry and align procedure you learned in school, and a carry would have to cross from the `4` chunk into the `102` chunk. Instead it leans on sums it happened to memorize from training. That is why it gets `2 + 2` right but stumbles on `47 × 83`. A practical fix is to let the model call a calculator or code interpreter for exact math.

The root cause is tokenization. cl100k_base caps digit runs at three digits, so it splits numbers at three-digit boundaries, **not** at place-value lines. The model rarely gets the one thing reliable arithmetic needs: a stable single-digit unit. Numbers of three digits or fewer like 512 can be a single token, but anything longer splits: `1024` is `['102', '4']`, `2048` is `['204', '8']`. The boundaries land at the three-digit cap, never at the thousands or hundreds line. Concrete example: 1024 + 999. In cl100k_base `1024` is `['102', '4']` and 999 is one token. So the model must produce 2023 without ever accessing the individual digits as separate quantities. A carry out of the ones place must cross from the `4` chunk into the `102` chunk. Now compare 1025 + 998: `1025` is `['102', '5']`, so the carry crosses the boundary differently and whatever pattern the model learned for the first case does not transfer. Multiplication compounds this. Digit by digit multiplication needs every place value, but if `47` is one token the model has no internal 4 and 7 to work with. It falls back on recalling memorized results, which is not a procedure. The practical mitigation is to route exact math to a code interpreter, or to space out digits like `4 7` so each becomes its own token.