What is semantic chunking and when does it beat fixed size chunking?

Flashcard·Medium·4.3 · 54·~30s

Attempt it

Semantic chunking splits a document at points where the embedding similarity between adjacent sentences drops, i.e., at natural topic boundaries, rather than at fixed token counts. It beats fixed size chunking when documents have heterogeneous content where rigid splits would cut mid-thought (long-form articles, mixed-topic reports). It loses when documents are short and uniform, where fixed size has no penalty.

Your answer

**Semantic chunking** is a way of splitting text into pieces by meaning rather than by token count. The algorithm walks through the document, embeds each sentence or short window, computes the embedding similarity between adjacent windows, and cuts wherever the similarity drops sharply. Drops mark places where the topic shifts. Fixed-size chunking is the simpler alternative. It just counts characters or tokens and cuts at a fixed length, ignoring what the text is actually about. This is cheap and predictable, but it sometimes cuts mid-sentence or mid-paragraph, producing chunks that mix two unrelated ideas. Semantic chunking wins when the documents are long and heterogeneous. Think of a research paper that includes background, method, results, and discussion in one file, or a customer support article that bundles unrelated topics. Fixed-size splits across these boundaries produce chunks whose embeddings are a confused average of two topics, which hurts retrieval. Fixed-size chunking is fine, and sometimes preferable, when documents are short, uniform, or already structured by clear headings. The extra cost of computing per-sentence embeddings to find boundaries does not pay off when there are no interesting boundaries to find.

**Semantic chunking** splits documents at points where adjacent-sentence embedding similarity drops below a threshold, which corresponds to topic shifts. Fixed-size chunking ignores content and cuts at a fixed token count. The algorithmic difference is small but consequential. Semantic chunking embeds each sentence, computes pairwise similarities along the sequence, and inserts a break wherever the rolling similarity drops below a tuned threshold. Fixed-size simply increments a token counter. Semantic chunking wins on long, heterogeneous documents where fixed splits would slice mid-thought. Examples include long-form articles, multi-topic reports, customer-support documentation with mixed subjects, and book-length material. The improvement shows up in top-k retrieval precision because each chunk's embedding describes a coherent thought. Fixed-size wins on short, uniform documents and on ingestion paths with tight latency budgets. Computing per-sentence embeddings at indexing time multiplies the embedding cost, sometimes by an order of magnitude, and that cost is wasted when documents lack interesting topic boundaries. The practical recommendation is to evaluate on a labelled retrieval benchmark for the actual corpus, since the answer depends heavily on what the documents look like rather than on theory.

Quick reference

TL;DR

Semantic chunking splits where adjacent-sentence embedding similarity drops, finding topic boundaries instead of token counts. It beats fixed-size chunking on heterogeneous documents and loses on short, uniform ones.

Memory aid

Easy to grasp

Imagine cutting a long radio show into clips. A clumsy way is to cut every two minutes regardless of what is being said. Some clips will start mid-sentence and stop mid-laugh. A thoughtful way is to listen, notice when the topic changes, and cut there. The clips end up varying in length, but each one is a coherent moment. Semantic chunking is the thoughtful way: it splits text at points where the meaning shifts, using embeddings to detect those shifts automatically. It is more work to set up, but each chunk ends up describing one thing. When the source is short and stays on a single topic, the clumsy cuts work fine and the extra thoughtfulness is wasted.

Key concepts

Chunking Strategies

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Define semantic chunking with the adjacent-sentence-similarity algorithm, contrast with fixed-size on both mechanism and outcome, then walk through when each one wins. Anchor the discussion on document heterogeneity as the dominant factor, mention the typical hybrid pattern, and close by noting that empirical evaluation on the actual corpus is the right way to decide.

Real-world examples

Real products, models, and research that use this idea.

LangChain's `SemanticChunker` uses adjacent-sentence cosine similarity drops with a configurable breakpoint percentile to find topic shifts.
LlamaIndex's `SemanticSplitterNodeParser` follows the same algorithmic idea with a buffered comparison window for noise reduction.
Anthropic's contextual retrieval skips semantic chunking entirely and instead prepends a generated summary to each chunk at indexing time, recovering the precision lift in a different way.
Production RAG stacks at companies like Notion and Glean typically combine structural splitting (headings, paragraphs) with semantic chunking only on the long sections that survive structural cuts.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you tune the similarity-drop threshold for a new corpus?

QWhy does hybridising structural and semantic splitters often outperform either alone?

QWhat benchmark would you use to compare two chunking strategies head-to-head?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating semantic chunking as universally better. The extra cost of computing per-sentence embeddings and similarity drops only pays back when documents are long, heterogeneous, and have real topic boundaries to find.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.