Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
The bug is on the last line: `return text[:limit]`. The docstring promises to truncate to a token limit, but `text[:limit]` keeps the first `limit` characters, not the first `limit` tokens.
In BPE tokenization a **token** is a subword unit, not a single character. A common word like 'running' might be one token spanning seven characters, while a rare word or an emoji might fragment into several tokens. So the character to token ratio is not fixed.
Because of that, slicing at character 100 produces an unpredictable token count. For plain English it might be 20 to 30 tokens; for code it might be 50 to 80; for Chinese it can run far higher. The function cannot honor its contract.
The correct pattern is encode, slice, decode: turn the text into token IDs with the tokenizer, keep the first `limit` IDs, then decode those back into a string. A second, subtler issue: a raw character slice can also cut a multi-byte character in half and corrupt the text.
The error is the last line: `return text[:limit]` slices by character position, but the function promises a token limit. In BPE, characters and tokens are not 1:1.
A single token spans roughly 1 to 6 characters depending on the word and the tokenizer. So 100 characters of English prose is maybe 20 to 30 tokens, while 100 characters of Python code can be 50 to 80, and non-Latin scripts can be higher still. The same `limit=100` yields a different token count for every content type, so the limit is never actually enforced.
The fix is the encode, slice, decode pattern:
```python
def truncate_to_token_limit(text, tokenizer, limit=100):
return tokenizer.decode(tokenizer.encode(text)[:limit])
```
Encode to token IDs, slice the ID array at `limit`, then decode. That guarantees exactly `limit` tokens.
There is also a secondary failure: a raw character slice can bisect a multi-byte character or grapheme cluster, producing invalid output or mojibake. This whole bug is extremely common in RAG chunking pipelines that assume characters approximate tokens.
Quick reference
TL;DR
The code slices by characters but promises a token limit. Tokens are not characters, so 100 chars can be 20 or 200 tokens. Fix: encode, slice IDs, decode.
Imagine a recipe says 'use the first 100 ingredients' but you instead grab the first 100 letters off the ingredient labels. Some ingredients have short names, some have long ones, so you end up with a wildly different number than 100 ingredients.
That is this bug. A token is more like a whole ingredient than a single letter. The word 'hello' is five letters but one token, while a long rare word might be many tokens. So cutting at letter 100 gives you an unpredictable number of tokens. The fix is to first turn the text into a list of token numbers, then keep the first 100 numbers, then turn them back into text.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Latency-critical paths where you only need a rough size guard, not an exact token cut Full encode and decode adds overhead; a conservative character heuristic with a safety buffer may be acceptable when exactness is not required.
Truncating with a tokenizer that does not match the serving model Encode, slice, decode is only correct when the tokenizer is the model's own; a mismatched one re-introduces a count error.
Splitting where semantic boundaries matter more than an exact token count Token slicing cuts mid-sentence; sentence- or structure-aware chunking with a token check is better for retrieval quality.
Streaming text where the full string is not yet available Encode, slice, decode assumes the whole input; incremental tokenization needs a running encoder that handles partial trailing bytes.
Quick scan · 2–4 min
Everything important, quickly.
2 min: char slice vs token limit + ratio varies by content + encode slice decode fix + Unicode boundary failure + RAG chunking context.
Code example
python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
# Buggy: character slice does not respect a token limit
def truncate_wrong(text: str, limit: int = 100) -> str:
return text[:limit]
# Correct: encode, slice the ID array, decode
def truncate_to_token_limit(text: str, limit: int = 100) -> str:
ids = tok.encode(text, add_special_tokens=False)
return tok.decode(ids[:limit], skip_special_tokens=True)
for t in ["Hello world! " * 10, "def f(x):return x*2\n" * 6, "这是中文测试" * 5]:
ids = tok.encode(t, add_special_tokens=False)
print(len(t), len(ids), round(len(ids) / len(t), 2))
Real-world examples
Real products, models, and research that use this idea.
1LangChain ships TokenTextSplitter alongside its character splitter precisely because character-based chunking overflowed model context windows.
2LlamaIndex's TokenTextSplitter uses the model-specific tokenizer so chunk sizes match the real token budget at ingestion time.
3Production RAG over CJK and Arabic corpora regularly blew past a GPT-5.5 context window when chunked by character count, because those scripts inflate token counts several-fold.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you implement a token-aware chunker that splits a document into N-token chunks with K-token overlap?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the cost of encode, slice, decode over a large corpus and how do you optimize it?
Sign in or go Pro to see the approach for this follow-up.
QWhen exact tokenization is too slow, what heuristic gives a safe token estimate?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming roughly 4 characters per token as a safe universal rule. It holds for English prose but fails for code (often 2:1) and non-Latin scripts (can be far worse).