Lost in the Middle
Models attend best to the start and end of long contexts and lose info in the middle.
A documented LLM failure mode where models pay disproportionate attention to information at the beginning and end of long contexts while neglecting the middle. Has direct implications for RAG ordering and long-context prompting.
In practice
Why RAG result ordering matters and why long-context windows don't automatically help. Standard senior-level gotcha question.
Related topics
Related terms
Retrieval-Augmented Generation (RAG)
Fetch relevant docs at inference time and stuff them into the prompt so the model can answer from real data.
Chunking
Cutting documents into retrievable pieces before embedding them.
Context Window
The max number of tokens a model can attend to at once.
Hallucination
When a model confidently makes up something that isn't true.
AI System Design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
HNSW (Hierarchical Navigable Small World)
A graph-based ANN index with log-time approximate search, excellent recall/latency trade-off.