Design a summarizer prompt that preserves user stated facts under aggressive compression

Short answer·Medium·4.0 · 0·~3 min·Asked atRunway Sap Sigmoid

Attempt it

Write the shape of a summarizer system prompt that compresses 20 turns to 500 tokens without dropping user stated facts. What instructions are essential? What is the failure mode you are designing against?

Free · 2 AI evals / day

A summarizer prompt is not just "compress this conversation in 500 tokens." That instruction lets the model decide what to keep, and the model will keep whatever makes the summary read fluently. User-stated facts are usually not what makes prose fluent, so they tend to be the first thing dropped. The fix is **structure**. Tell the summarizer to produce a sectioned output, not free-form prose. A good shape has four sections: - **Durable user facts**: things the user stated that should survive across the whole conversation (allergies, preferences, identifiers, hard constraints). - **Open items**: questions or tasks the user is still waiting on. - **Completed items**: tasks already done, with their key outcomes. - **Narrative**: a brief paragraph tying the conversation together. Each section gets its own token cap. The durable-facts section, say, gets 150 tokens of its own and is forbidden from being trimmed to make room for narrative. Three instructions are essential. First, **quote user statements verbatim** rather than paraphrasing. Second, **omit polite chatter and acknowledgments entirely**. Third, **never invent** details that did not appear in the turns. The failure mode being designed against is fact loss disguised as polished prose: a beautiful summary that no longer remembers the user said "I am allergic to peanuts."

The prompt should produce a **sectioned** output rather than free-form prose, because free-form prose lets the model trade off user-stated facts against narrative polish, and polish wins. Four sections cover the working set. **Durable user facts** holds things the user stated that should survive across the whole conversation: allergies, preferences, identifiers, hard constraints, and decisions. **Open items** holds questions or tasks still pending. **Completed items** holds tasks already done with their key outcomes. **Narrative** is a brief paragraph tying the conversation together. Each section gets its own protected token budget. The durable-facts section might own 150 tokens of the 500 total and is forbidden from being trimmed when narrative wants more space. This is the key design move: without per-section caps, the summarizer trades facts for fluency at every refresh. Four instructions are essential to the prompt body. First, **quote user-stated facts verbatim** rather than paraphrase. The summarizer should copy the user's words inside the facts section, especially for numerics, names, and constraints. Second, **omit polite chatter and acknowledgments entirely**, since they contribute nothing across refreshes. Third, **flag any contradictions** the summarizer spots between turns, so the downstream model can resolve them rather than silently averaging. Fourth, **never invent details** that did not appear in the source turns. The failure mode being designed against is fact loss disguised as polished prose. A free-form summary at 500 tokens reads beautifully, scores well on semantic-similarity evals, and quietly no longer mentions that the user said "I am allergic to peanuts." Sectioning, per-section caps, and the verbatim-quote rule are the structural defenses against that failure.

TL;DR

Use a sectioned summary with a protected user-facts block, per-section token caps, and a verbatim-quote rule for stated facts. The failure mode is fluent prose that quietly drops the one fact the user most cared about.

Memory aid

Easy to grasp

Picture a hotel concierge who takes notes about a guest. A bad concierge writes one paragraph: "the guest seems easy-going and likes Italian food." A good concierge fills out a card with labeled boxes: name, room number, allergies, preferences, open requests. The labeled boxes mean nothing critical gets forgotten just because the concierge ran out of room. A summarizer prompt works the same way. If you ask for free-form prose, the summary will read smoothly but the allergy might silently disappear. If you ask for labeled sections with a protected facts box, the allergy survives every refresh because there is a named place for it to live.

Key concepts

Context Engineering Agents

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A summarizer prompt is one of the highest-leverage pieces of text in any production chat or agent system. The prompt decides which user statements survive a refresh, which fade, and which silently disappear. A poorly designed prompt can pass every aggregate eval (semantic similarity, ROUGE, BLEU) while quietly destroying the application's ability to remember the one fact each user most cares about.

This deep dive walks through the design of a summarizer prompt that compresses 20 turns to 500 tokens without dropping user-stated facts. The shape generalizes to other compression ratios and other domains; the principles do not change.

The framing: structured extraction, not paraphrase

The first design move is to stop thinking of the summarizer as a paraphraser and start thinking of it as a structured-extraction step that also produces narrative. A paraphraser is asked "say the same thing in fewer words." An extractor is asked "pull these specific kinds of content into these specific slots, and then write a brief narrative for the rest."

The difference is not stylistic. A paraphraser has full latitude to drop whatever it thinks is least important to the overall meaning. User-stated facts (a budget figure, an allergy, an identifier) typically contribute little to the overall meaning of a conversation but are exactly the things the application cannot afford to lose. A paraphraser, doing its job well, drops them.

An extractor cannot drop them, because the prompt explicitly names a slot for them and forbids the slot from being empty or paraphrased. The same model with the same input produces a fundamentally different output when the framing changes from "summarize" to "extract into these labeled slots, then narrate the rest."

This framing also makes the output evaluable. A free-form summary is evaluated with fuzzy metrics that average across all the content. A sectioned summary can be evaluated section by section: the facts section gets exact-match recall, the narrative section gets semantic similarity. Different sections, different metrics, different defenses.

The four sections and their per-section caps

The five operational instructions in the prompt body

The failure mode this design defeats

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Mem0 implements the durable-facts idea as a separate extracted-fact store, sidestepping prose summarization for those specifics.
Anthropic's Claude projects expose a custom-instructions block that effectively carries verbatim user-stated facts across sessions.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you measure whether your summarizer prompt is actually preserving user-stated facts?

Build a small offline eval that injects synthetic user-stated facts into a 20-turn synthetic conversation, runs the summarizer, and checks whether each fact appears verbatim in the output. Score is per-fact recall, not overall similarity. Run the eval on every prompt change before shipping.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Asking the summarizer for free-form prose with a total token budget. Without per-section caps and a protected facts block, the model trades user facts for narrative polish.

60 second bullets to scan on the way to the call.

List the four sections of a durable summarizer prompt
Explain why sectioning beats free-form prose under aggressive compression

Primary sources. Browse if you want the original framing.