Pick the soft-budget allocation pattern that survives a long RAG conversation

Attempt it

Quick reference

TL;DR

Reserve floors for the non-negotiable slots (system, memory, schema) and let retrieval and history compete for the residual pool under an explicit eviction policy.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine packing a small suitcase for a week-long trip. You always need a toothbrush, your passport, and one warm layer, those go in first, every trip. The remaining space is contested by clothes, books, and snacks, and you have a rule for what gets pulled when the zipper will not close. Maybe books lose first, then snacks, then a second outfit. A context budget works the same way. Some slots are non-negotiable: the system prompt, user memory, the output schema. Those get protected floors. Everything else, retrieved documents, older chat history, competes for the leftover room, and you write down what gets cut first when the suitcase fills.

Key concepts

Context Engineering Retrieval-Augmented Generation Prompt Engineering

Quick scan · 2–4 min

Everything important, quickly.

6 minutes: why fixed splits fail, the three-tier structure, the contested pool and its weights, and the explicit eviction order.

Code example

python

def allocate_budget(total: int, slots: dict, signals: dict) -> dict:
    # Tier 1: protected floors
    floors = {
        'system': slots['system_size'],
        'memory': slots['memory_size'],
        'schema': slots['schema_size'],
        'response': 1024,  # response reserve
    }
    floor_total = sum(floors.values())
    residual = total - floor_total
    if residual < 0:
        raise BudgetError('floors exceed total budget')

    # Tier 2: contested pool, signal-weighted
    retrieval_weight = max(0.3, min(0.8, signals['rerank_top1']))
    history_weight = 1.0 - retrieval_weight
    contested = {
        'retrieval': int(residual * retrieval_weight),
        'history':   int(residual * history_weight),
    }
    return {**floors, **contested}

# Eviction policy: oldest summary first, then lowest-rerank chunk
def evict(over_by: int, packed: dict) -> dict:
    while over_by > 0:
        if packed['history_summary_tail']:
            over_by -= drop_oldest_summary(packed)
        elif packed['retrieval_chunks']:
            over_by -= drop_lowest_rerank(packed)
        else:
            zero_out_bleed_pool(packed)
            break
    return packed

Real-world examples

Real products, models, and research that use this idea.

Anthropic's Claude tool-use docs recommend keeping the system block and tool definitions in a stable cached prefix while allowing the user turn and retrieval to flex per call.
LangChain's `ConversationSummaryBufferMemory` and LlamaIndex's `ContextChatEngine` both implement variants of priority-tiered budgeting with explicit eviction order.
Cursor's agent context allocator reserves slots for active file content and tool schemas, with retrieval and chat history competing for the residual against an explicit cap.
Perplexity's search-grounded answer stack uses rerank scores to drive how many chunks each turn gets, rather than a fixed top-k.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide the floor sizes for a new application?

Sign in or go Pro to see the approach for this follow-up.

QIf retrieval keeps starving history, how do you tell whether the weight is wrong or the rerank is wrong?

Sign in or go Pro to see the approach for this follow-up.

QHow should the allocator behave when prompt caching is on?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.