Semantic boundaries, downstream context size, and embedding latency are real chunking decisions. The LLM tokenizer's scheme and vocabulary size are not chunking concerns at all.
Picture cutting a long book into bookmarks for a librarian who only ever reads the bookmarked passages. You care about where you cut, ideally between chapters so each bookmark is a coherent thought. You care about how big each cut is, since the librarian can only hold a few at a time. You care about how long it takes to bookmark every page, since you might have a million pages. You do not care which font the printer used to typeset the words. The font is the tokenizer. It matters when the words get read out loud later, but it does not change where you should cut the book.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Chunking is the unglamorous decision that sets the ceiling on RAG quality. The retriever cannot fix bad chunks, and the LLM cannot reason over fragments. A team that gets chunking right unlocks gains the rest of the pipeline cannot deliver on its own.
This question asks you to separate real chunking tradeoffs from distractors that sound plausible because they involve text and tokens, but operate at a different layer entirely. Each of the three correct options ties to a real axis of the design space. Each of the two distractors confuses chunking with downstream tokenisation.
Why semantic boundaries are a real concern
Embeddings describe what a chunk is about. A chunk that contains one coherent thought produces an embedding concentrated in the right region of semantic space. A chunk that cuts across two unrelated thoughts produces an embedding that averages between them, which lowers precision because it neither matches one topic strongly nor stands clear of the other.
Fixed-size splitters that respect natural boundaries, like markdown headers, paragraphs, or sentence breaks, recover most of this lift without much complexity. Semantic chunking, which splits where adjacent-sentence similarity drops below a threshold, recovers more but at the cost of computing similarities during ingestion.
The tradeoff is between splitter complexity and retrieval precision. For homogeneous, well-structured corpora the marginal gain from semantic chunking is small. For mixed, heterogeneous content like long reports and articles with embedded code, semantic methods often pay back their cost in measurably better top-k recall.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LangChain's `RecursiveCharacterTextSplitter` is a popular default that respects markdown and paragraph boundaries while honouring a target size, hitting axes A and B simultaneously.
- LlamaIndex's `SemanticSplitterNodeParser` splits on embedding-similarity drops between adjacent sentences, directly optimising for semantic boundaries.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you decide between fixed-size and semantic chunking for a corpus of regulatory filings?
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating tokenizer details as chunking concerns. Chunks are stored as text or pre-embedded vectors. The downstream model retokenises whatever context wins the retrieval, so the tokenizer choice does not shape chunk design.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.