Defend a token budget split between rolling summary, recent turns, and retrieved context for a support chatbot

Short answer·Medium·4.0 · 0·~3 min·Asked atCitadel Cohere Flowise

Attempt it

You have a 16k-token window for a customer support chatbot. Propose a budget split across system prompt, persistent memory, rolling summary, recent verbatim turns, and retrieved KB chunks. Justify the priorities.

Free · 2 AI evals / day

A 16k-token window for a customer-support chatbot has to feed several distinct slots. The defensible allocation starts from what is non-negotiable and works inward. **Protected slots first.** The system prompt (persona, capabilities, rules) needs 600-1000 tokens; allocate 800. The output reservation, how many tokens the model needs for its answer, needs 1500-2500 for a typical support response; allocate 2000. These two are fixed; everything else negotiates around them. **Content slots next.** Retrieved KB chunks get the biggest share because the task is grounded in documentation; allocate 6000 (enough for 4-6 reranked chunks at typical chunk size). Recent verbatim conversation turns get the next biggest slot at 4000 because verbatim recency carries pronouns, exact strings, and the latest user intent, the highest-signal context the model has. **Lower-priority slots.** Rolling summary gets 1200, its job is connective tissue, not the main signal. Persistent memory (cross-session user facts) gets 600; it is small but high-priority because it personalises the answer. **Slack.** 1400 tokens of slack absorbs day to day variation and keeps prompt caching stable. Total: 800 + 600 + 1200 + 4000 + 6000 + 2000 + 1400 = 16000.

**Defensible split for 16k tokens, customer-support chatbot:** - System prompt: **800** - Persistent memory (cross-session user facts): **600** - Rolling summary (older turns compressed): **1200** - Recent verbatim conversation turns: **4000** - Retrieved KB chunks: **6000** - Output reservation: **2000** - Slack: **1400** The allocation order matters more than the exact numbers, and it goes from least-flexible to most-flexible. **Output reservation is non-negotiable.** An answer that truncates mid-sentence is a worse failure than any retrieval degradation. 2000 tokens covers the verbose but not padded answers support chatbots produce. **KB chunks get the largest content slot** because support tasks are grounded in documentation. 6000 tokens supports 4-6 reranked chunks at typical chunk size. **Recent verbatim turns beat rolling summary.** Verbatim recency carries pronouns, exact strings, and current user intent, information that a summary cannot reconstruct without loss. 4000 tokens covers the last 6-10 turns at typical density. **Rolling summary is connective tissue.** 1200 tokens covers the arc before the verbatim window. It keeps context coherent, not to be the main signal. **Persistent memory is small but high-value.** 600 tokens for user-level facts personalises the answer across sessions. **Slack absorbs variation.** 1400 tokens keeps the budget stable against chunk-size variance and reduces prompt-cache misses from prefix shifts. Eviction order under pressure: shrink KB chunks first (drop top-5 to top-3), then rolling summary, then oldest verbatim turns. Never the output reservation.

TL;DR

Protect output reservation and system prompt first, allocate the biggest content slot to retrieval, give recent turns more space than rolling summary, and keep slack to avoid prompt-cache misses.

Memory aid

Easy to grasp

Picture packing for a flight with a hard weight limit on the bag. Some things are fixed and non-negotiable: the boarding pass, your passport, the clothes you will wear off the plane. Those go in first and you do not weigh them again. After that, you allocate the remaining weight by how badly you need each thing. The textbook for the trip gets a lot because it is the whole point. A small notebook of personal preferences earns a little space because it changes every trip. Old itineraries from last year get a single summary card, not the full paper. And you leave a little slack so the bag still closes when you stuff something in at the gate. A token budget is the same shape.

Key concepts

Context engineering Retrieval-augmented generation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A context-window budget is a multi-objective allocation problem. Each slot competes for the same finite token pool, but the slots are not interchangeable. Some carry the dominant signal for the task; others are scaffolding; some are non-negotiable. The art is in the priority order, not the exact numbers.

This card walks through a defensible split for a 16k-token customer-support chatbot, the reasoning behind each allocation, and how to evict under budget pressure without breaking the answer.

The four-tier priority

Every context-window allocation falls into one of four priority tiers. The tiers are about flexibility under budget pressure, not about absolute size.

Tier 1: Protected. Slots that cannot float. Output reservation and system prompt. The output slot is sized by the task floor, the smallest answer the system has to produce without truncation. The system prompt is sized by the rules the model must follow. Both are roughly fixed for a given application.

Tier 2: Signal-carrying. Slots that carry the dominant information for the task. For a grounded support chatbot, that is retrieved KB chunks and recent verbatim conversation turns. These two get the biggest share because the answer literally depends on what is in them.

Tier 3: Scaffolding. Slots that provide coherence and personalisation but are not the main signal. Rolling summary (connective tissue for older conversation) and persistent memory (cross-session user facts). These get modest budgets because their value is in being present at all, not in being large.

Tier 4: Slack. Tokens reserved for variation and prompt-cache stability. Not a content slot; the absence of slack causes production failures even when the content slots are sized correctly.

Why the order matters

When the budget tightens, you evict from the bottom of the priority list. The order determines what the system degrades to under load, which is what production correctness depends on.

Output reservation: the protected floor

The signal-carrying tier: KB chunks and recent turns

The scaffolding tier: summary and memory

Slack, eviction, and production tuning

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's Claude Opus 4.7 support-chatbot reference architecture uses a similar split: large retrieval slot, generous output reservation, modest rolling summary.
LangGraph's MemorySaver-plus-retrieval pattern for production support agents explicitly separates persistent memory from rolling summary as different stores.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you change this split for a 200K-token window?

Output reservation and system prompt stay similar; retrieval can stretch to top-10 or top-15 chunks; verbatim turns can carry more history without needing summarization; rolling summary becomes optional for moderate-length conversations.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Splitting the budget evenly across all slots, or letting output reservation float. Output tokens are fixed by the task, not negotiable against input; treating them as flexible is how production prompts truncate mid-answer.

60 second bullets to scan on the way to the call.

Which slots are protected and why output reservation tops the list
Why KB chunks get the largest content slot for a grounded task

Primary sources. Browse if you want the original framing.