Defend a token-budget split between rolling summary, recent turns, and retrieved context for a support chatbot

Attempt it

You have a 16k-token window for a customer-support chatbot. Propose a budget split across system prompt, persistent memory, rolling summary, recent verbatim turns, and retrieved KB chunks. Justify the priorities.

Free · 2 AI evals / day

A 16k-token window for a customer-support chatbot has to feed several distinct slots. The defensible allocation starts from what is non-negotiable and works inward. **Protected slots first.** The system prompt (persona, capabilities, rules) needs 600-1000 tokens; allocate 800. The output reservation, how many tokens the model needs for its answer, needs 1500-2500 for a typical support response; allocate 2000. These two are fixed; everything else negotiates around them. **Content slots next.** Retrieved KB chunks get the biggest share because the task is grounded in documentation; allocate 6000 (enough for 4-6 reranked chunks at typical chunk size). Recent verbatim conversation turns get the next biggest slot at 4000 because verbatim recency carries pronouns, exact strings, and the latest user intent, the highest-signal context the model has. **Lower-priority slots.** Rolling summary gets 1200, its job is connective tissue, not the main signal. Persistent memory (cross-session user facts) gets 600; it is small but high-priority because it personalises the answer. **Slack.** 1400 tokens of slack absorbs day to day variation and keeps prompt caching stable. Total: 800 + 600 + 1200 + 4000 + 6000 + 2000 + 1400 = 16000.

**Defensible split for 16k tokens, customer-support chatbot:** - System prompt: **800** - Persistent memory (cross-session user facts): **600** - Rolling summary (older turns compressed): **1200** - Recent verbatim conversation turns: **4000** - Retrieved KB chunks: **6000** - Output reservation: **2000** - Slack: **1400** The allocation order matters more than the exact numbers, and it goes from least-flexible to most-flexible. **Output reservation is non-negotiable.** An answer that truncates mid-sentence is a worse failure than any retrieval degradation. 2000 tokens covers the verbose but not padded answers support chatbots produce. **KB chunks get the largest content slot** because support tasks are grounded in documentation. 6000 tokens supports 4-6 reranked chunks at typical chunk size. **Recent verbatim turns beat rolling summary.** Verbatim recency carries pronouns, exact strings, and current user intent, information that a summary cannot reconstruct without loss. 4000 tokens covers the last 6-10 turns at typical density. **Rolling summary is connective tissue.** 1200 tokens covers the arc before the verbatim window. It keeps context coherent, not to be the main signal. **Persistent memory is small but high-value.** 600 tokens for user-level facts personalises the answer across sessions. **Slack absorbs variation.** 1400 tokens keeps the budget stable against chunk-size variance and reduces prompt-cache misses from prefix shifts. Eviction order under pressure: shrink KB chunks first (drop top-5 to top-3), then rolling summary, then oldest verbatim turns. Never the output reservation.

Quick reference

TL;DR

Protect output reservation and system prompt first, allocate the biggest content slot to retrieval, give recent turns more space than rolling summary, and keep slack to avoid prompt-cache misses.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture packing for a flight with a hard weight limit on the bag. Some things are fixed and non-negotiable: the boarding pass, your passport, the clothes you will wear off the plane. Those go in first and you do not weigh them again. After that, you allocate the remaining weight by how badly you need each thing. The textbook for the trip gets a lot because it is the whole point. A small notebook of personal preferences earns a little space because it changes every trip. Old itineraries from last year get a single summary card, not the full paper. And you leave a little slack so the bag still closes when you stuff something in at the gate. A token budget is the same shape.

Key concepts

Context engineering Retrieval-augmented generation

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 minutes: the four-tier priority (protected, signal, scaffolding, slack), why recent turns beat summary, why KB is the largest slot, eviction order, slack and caching, tuning levers.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's Claude Opus 4.7 support-chatbot reference architecture uses a similar split: large retrieval slot, generous output reservation, modest rolling summary.
LangGraph's MemorySaver-plus-retrieval pattern for production support agents explicitly separates persistent memory from rolling summary as different stores.
Mem0 and Letta both treat persistent memory as a high-priority small slot that survives across sessions, matching the 600-token allocation pattern.
Cohere's 2026 RAG cookbook for grounded chat use cases puts retrieval at 30-40 percent of the window for the same reasons.
Pinecone's customer-support sample app templates ship with a default budget allocation in the same shape (large KB slot, generous recent-turn slot, modest summary).

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you change this split for a 200K-token window?

Sign in or go Pro to see the approach for this follow-up.

QWhat happens if persistent memory grows past 600 tokens for a heavy user?

Sign in or go Pro to see the approach for this follow-up.

QHow does prompt caching influence the allocation?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Splitting the budget evenly across all slots, or letting output reservation float. Output tokens are fixed by the task, not negotiable against input; treating them as flexible is how production prompts truncate mid-answer.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.