What's the practical distinction between 'the prompt' and 'the context' in an LLM API call?

**Prompt** and **context** are useful words for humans, but the model does not see a difference between them. Everything you send in an API call (system message, user message, retrieved documents, tool outputs, previous turns) is one flat sequence of tokens. The model reads that sequence and produces output one token at a time. There is no second store of information called the context window that lives separately from the prompt. When developers say prompt, they usually mean the parts they authored: the instructions, the output format, the few-shot examples, the role. When they say context, they usually mean the parts that change per call: the retrieved documents from RAG, the user's question, the current conversation state. Both end up as input tokens, and attention treats them the same way. This matters in practice because it tells you that placement, ordering, and length all count, regardless of which conceptual bucket the content belongs to.

**The distinction collapses at the model level.** Everything you send in an API call (system message, user messages, retrieved documents, tool outputs, conversation history) gets concatenated into one input sequence. The attention layer treats it uniformly. There is no architectural boundary between prompt tokens and context tokens. The split is a human convention. Developers say prompt for the part they author and ship as code: instructions, output schema, few-shot examples, role definition, refusal rules. They say context for the part that varies per call: retrieved chunks from a vector store, the user's question, conversation state, tool-call results. Both occupy the same input stream and consume the same input-token budget. The practical consequences are real. Placement matters because long-context LLMs show U-shape attention bias. Token budget is shared, so a bloated system prompt eats the same budget as retrieved chunks. Prompt caching at hosted providers reuses a stable prefix, which is why the convention puts authored content (the prompt) at the top and varying content (the context) at the bottom. The distinction is for humans to reason about, not something the model itself respects.