Spot the context-engineering bug in this evidence-block assembly

Spot the error·Medium·4.0 · 0·~2 min·Asked atCohere Anthropic OpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Two things are wrong in the description. The first error is sorting the chunks by document title alphabetically. Alphabetical order has no relationship to how useful a chunk is for the current query. The most-relevant chunk might end up at position three or four, where the lost-in-the-middle effect kicks in and the model attends to it less. The whole point of having a retrieval ranker is to put the most useful chunk where the model is most likely to use it. The second error is skipping per-chunk identifiers. The reasoning given, that the model can just quote chunk text in the citation, is exactly the failure mode the identifier pattern exists to prevent. When the model has to produce a citation by quoting or paraphrasing chunk text, it confabulates. It mangles titles, invents URLs, and the renderer has no reliable way to link the cite back to the original source row. The fix is twofold. Sort by relevance score, usually the reranker score from a cross-encoder, and either head-load the strongest chunk or use a head and tail pattern if you are exploiting position bias. And attach a stable identifier like [S1], [S2] inline with each chunk so the model can cite the id and the renderer can map it back to the source.

The evidence-block assembly described has two compounding bugs that together make both lost-in-the-middle mitigation and citation linkage fail. The first bug is **alphabetical sort by document title**. Title order is uncorrelated with relevance, so the strongest chunk lands at a random position in the block. That is the worst case for the well-documented lost-in-the-middle curve, which shows models attending more to content at the start and end of a long context block than to content in the middle. The whole point of running a reranker (or even a basic similarity sort) is to get useful chunks into privileged positions; alphabetical sort throws that away. The second bug is **no per-chunk identifier**. The justification, that the model can quote chunk text in its citation, is a known failure mode. When the model is forced to construct a citation by quoting or paraphrasing, it confabulates. It paraphrases titles wrong, invents plausible-looking URLs, and the render-time mapping back to source rows breaks. The fixes are independent and both required. For ordering, sort by relevance score from the reranker; if position bias matters, place the strongest chunk at the head and the second-strongest at the tail. For citations, attach a stable id (chunk_id or short tag like [S1], [S2]) inline with each chunk as a header line, and instruct the system prompt to cite by tag.

Quick reference

TL;DR

Two compounding bugs, alphabetical ordering throws away the relevance signal, and skipping per-chunk ids forces the model to confabulate citations from chunk text.

Memory aid

Easy to grasp

Picture handing a kid five flashcards to study for a quiz. Two mistakes here. First, you sorted the cards alphabetically by which book they came from, instead of putting the most useful card first or last where the kid will actually remember it. Second, you ripped the corner stickers off the cards, so when the kid says which card they used they have to describe what was written on it, and they will get the description wrong. Fix one: put the strongest card at the start, the second strongest at the end. Fix two: leave the stickers on so the kid can just say S1 or S2.

Key concepts

Context engineering Retrieval-augmented generation

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 minutes: identify both errors, explain why alphabetical sort interacts badly with lost-in-the-middle, describe the stable-id assembly pattern, name a provider-native citation API.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's 2026 RAG cookbook for Claude Opus 4.7 assembles evidence blocks with [S1]-style tags and orders by Cohere Rerank 3.5 score.
OpenAI's Responses API for GPT-5 ships first-class document-id citations, the SDK builds the assembly with stable ids by default.
Cohere Command R+ takes a `documents` parameter so the API itself enforces stable id to source binding.
LangGraph's RAG templates use a relevance-sorted assembler with explicit per-chunk ids and warn against title sorting.
Pinecone's hosted RAG starter explicitly head-loads the top-scoring chunk and tail-loads the second-best to exploit primacy and recency.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide between head-loading the top chunk versus head and tail loading?

QWhat does a render-time validator do with citations the model emits?

QHow does the assembly change for multi-turn conversations where prior turns also cited evidence?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Sorting evidence blocks by document title rather than by relevance score. Useful chunks land in lost-in-the-middle positions and the model misses them.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.