**Prompt** and **context** are useful words for humans, but the model does not see a difference between them. Everything you send in an API call (system message, user message, retrieved documents, tool outputs, previous turns) is one flat sequence of tokens. The model reads that sequence and produces output one token at a time. There is no second store of information called the context window that lives separately from the prompt.
When developers say prompt, they usually mean the parts they authored: the instructions, the output format, the few-shot examples, the role. When they say context, they usually mean the parts that change per call: the retrieved documents from RAG, the user's question, the current conversation state. Both end up as input tokens, and attention treats them the same way.
This matters in practice because it tells you that placement, ordering, and length all count, regardless of which conceptual bucket the content belongs to.
**The distinction collapses at the model level.** Everything you send in an API call (system message, user messages, retrieved documents, tool outputs, conversation history) gets concatenated into one input sequence. The attention layer treats it uniformly. There is no architectural boundary between prompt tokens and context tokens.
The split is a human convention. Developers say prompt for the part they author and ship as code: instructions, output schema, few-shot examples, role definition, refusal rules. They say context for the part that varies per call: retrieved chunks from a vector store, the user's question, conversation state, tool-call results. Both occupy the same input stream and consume the same input-token budget.
The practical consequences are real. Placement matters because long-context LLMs show U-shape attention bias. Token budget is shared, so a bloated system prompt eats the same budget as retrieved chunks. Prompt caching at hosted providers reuses a stable prefix, which is why the convention puts authored content (the prompt) at the top and varying content (the context) at the bottom. The distinction is for humans to reason about, not something the model itself respects.
Quick reference
TL;DR
At the model level, prompt and context collapse into one input stream the attention layer treats uniformly; the split is a useful human convention, not an architectural boundary.
Imagine handing a chef a single sheet of paper before they cook for you. The top of the page has your standing rules (no nuts, low salt, plate it cold). The bottom has tonight's order (mushroom risotto for two). When the chef reads the paper, they read the whole thing as one sheet. They do not have a separate 'rules folder' in their head and a 'tonight's order folder'. We call the top half 'the prompt' because that's the part you authored, and the bottom half 'the context' because it changes every night, but the chef just reads paper. LLMs do exactly the same thing with the input you send.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Explaining LLM behavior to a non-engineering stakeholder The prompt vs context split is a useful mental model for design conversations; collapsing it too early confuses the audience and hides intent.
Reasoning about prompt caching strategy at the boundary Caching depends on prefix stability; here the prompt vs context split is genuinely critical for billing and latency, not just convention.
Diagramming a RAG architecture for a design review The split helps reviewers see what is authored vs retrieved; insisting they collapse is technically true but hurts communication.
Quick scan · 2–4 min
Everything important, quickly.
3 min: one flat input stream + role markers as special tokens + shared token budget + caching as a convention enforcer + where the line blurs.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's prompt-caching feature on Claude Opus 4.7 caches a stable prefix (system + tools + few-shot) and re-runs only the tail (user query + retrieved chunks), which is exactly the prompt vs context split made into a billing optimization.
2OpenAI's GPT-5.5 input structure treats system, user, and assistant messages as one concatenated input under the hood; role markers are special tokens, not separate channels.
3LangGraph agents thread a single state object through every node; what the model receives at each step is one merged input regardless of how the developer organizes it in code.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does prompt caching change where you put things?
Sign in or go Pro to see the approach for this follow-up.
QDo role markers (system, user, assistant) carry any architectural weight?
Sign in or go Pro to see the approach for this follow-up.
QHow does this view change for multimodal inputs?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing the LLM has a separate context window for retrieved data, or that the model knows which tokens were authored by you versus pulled from a database.