What's the practical distinction between 'the prompt' and 'the context' in an LLM API call?
At the model level, prompt and context collapse into one input stream the attention layer treats uniformly; the split is a useful human convention, not an architectural boundary.
Imagine handing a chef a single sheet of paper before they cook for you. The top of the page has your standing rules (no nuts, low salt, plate it cold). The bottom has tonight's order (mushroom risotto for two). When the chef reads the paper, they read the whole thing as one sheet. They do not have a separate 'rules folder' in their head and a 'tonight's order folder'. We call the top half 'the prompt' because that's the part you authored, and the bottom half 'the context' because it changes every night, but the chef just reads paper. LLMs do exactly the same thing with the input you send.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prompt and context are two of the most-used words in LLM engineering, and they get used loosely. People say prompt when they mean the whole input, and they say context when they mean retrieved chunks specifically. Both habits are fine in casual conversation. They get in the way when you start optimizing real systems, because the model's behavior is shaped by where tokens sit, not by which conceptual bucket a human assigned them to.
This question asks for the practical distinction. The honest answer is that there is no architectural distinction at the model level. There is, however, a useful conceptual distinction that maps onto how you author, test, monitor, and bill for LLM systems. The deep dive separates the two views and walks through where each one earns its keep.
The goal is to leave you with a clean mental model: one input stream from the model's point of view, two human conventions on top, with clear rules for when each view should drive your decisions.
What the model actually sees
A decoder-only LLM takes a flat sequence of tokens, runs them through embedding + positional encoding, and applies causal self-attention layer by layer. The attention pattern at any given position can look back at every earlier token in the sequence. That is the only structural fact about how input gets processed.
Role markers such as system, user, assistant are not separate channels. They are special tokens (or special token sequences) inserted by the chat-template layer of the tokenizer. The model learned during instruction tuning to weight content that follows a system marker differently from content that follows a user marker, but that weighting is behavioral, not architectural. There is no separate attention head dedicated to system content. There is no separate context window. There is one window, and one attention pattern, across the whole input.
The same applies to retrieved documents, tool outputs, and conversation history. Whatever you concatenate into the input becomes part of the one stream the model attends over.
The Q, K, V matrices project the whole input. There is no carve-out for retrieved chunks vs authored instructions.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's prompt-caching feature on Claude Opus 4.7 caches a stable prefix (system + tools + few-shot) and re-runs only the tail (user query + retrieved chunks), which is exactly the prompt vs context split made into a billing optimization.
- OpenAI's GPT-5.5 input structure treats system, user, and assistant messages as one concatenated input under the hood; role markers are special tokens, not separate channels.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does prompt caching change where you put things?
Stable prefix at the top is cacheable and cheap; volatile content at the bottom; place the cache boundary so reuse is maximized across calls.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing the LLM has a separate context window for retrieved data, or that the model knows which tokens were authored by you versus pulled from a database.
60 second bullets to scan on the way to the call.
Why the model sees one input stream
Role markers as special tokens, not separate channels
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.