Spot the context engineering bug in this evidence block assembly

Spot the error·Medium·4.0 · 0·~2 min·Asked atIntel Stability Ai Turing

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Two things are wrong in the description. The first error is sorting the chunks by document title alphabetically. Alphabetical order has no relationship to how useful a chunk is for the current query. The most-relevant chunk might end up at position three or four, where the lost-in-the-middle effect kicks in and the model attends to it less. The whole point of having a retrieval ranker is to put the most useful chunk where the model is most likely to use it. The second error is skipping per-chunk identifiers. The reasoning given, that the model can just quote chunk text in the citation, is exactly the failure mode the identifier pattern exists to prevent. When the model has to produce a citation by quoting or paraphrasing chunk text, it confabulates. It mangles titles, invents URLs, and the renderer has no reliable way to link the cite back to the original source row. The fix is twofold. Sort by relevance score, usually the reranker score from a cross-encoder, and either head-load the strongest chunk or use a head and tail pattern if you are exploiting position bias. And attach a stable identifier like [S1], [S2] inline with each chunk so the model can cite the id and the renderer can map it back to the source.

The evidence-block assembly described has two compounding bugs that together make both lost-in-the-middle mitigation and citation linkage fail. The first bug is **alphabetical sort by document title**. Title order is uncorrelated with relevance, so the strongest chunk lands at a random position in the block. That is the worst case for the well-documented lost-in-the-middle curve, which shows models attending more to content at the start and end of a long context block than to content in the middle. The whole point of running a reranker (or even a basic similarity sort) is to get useful chunks into privileged positions; alphabetical sort throws that away. The second bug is **no per-chunk identifier**. The justification, that the model can quote chunk text in its citation, is a known failure mode. When the model is forced to construct a citation by quoting or paraphrasing, it confabulates. It paraphrases titles wrong, invents plausible-looking URLs, and the render-time mapping back to source rows breaks. The fixes are independent and both required. For ordering, sort by relevance score from the reranker; if position bias matters, place the strongest chunk at the head and the second-strongest at the tail. For citations, attach a stable id (chunk_id or short tag like [S1], [S2]) inline with each chunk as a header line, and instruct the system prompt to cite by tag.

TL;DR

Two compounding bugs, alphabetical ordering throws away the relevance signal, and skipping per-chunk ids forces the model to confabulate citations from chunk text.

Memory aid

Easy to grasp

Picture handing a kid five flashcards to study for a quiz. Two mistakes here. First, you sorted the cards alphabetically by which book they came from, instead of putting the most useful card first or last where the kid will actually remember it. Second, you ripped the corner stickers off the cards, so when the kid says which card they used they have to describe what was written on it, and they will get the description wrong. Fix one: put the strongest card at the start, the second strongest at the end. Fix two: leave the stickers on so the kid can just say S1 or S2.

Key concepts

Context engineering Retrieval-augmented generation

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The described assembly compounds two independent errors. Each one would be a quality regression on its own; together they produce a context block that is both poorly ordered for the lost-in-the-middle curve and broken for citation linkage.

This card pulls the two errors apart, explains why each one fails, and walks through what a correct evidence-block assembly looks like in a 2026 production stack.

Error one, sorting by document title

Alphabetical sort by document title has no relationship to query relevance. Whatever ranking the upstream retriever and reranker produced gets discarded when the assembler imposes title order.

The consequence interacts directly with the lost-in-the-middle effect documented by Liu et al. 2023. Models attend most strongly to content at the start and end of a long context block and least to content in the middle. The empirical drop on retrieval and reasoning tasks when a critical chunk lands in the middle is consistently in the 10-30 percent range across modern frontier models and across the RULER and BABILong long-context benchmarks.

Why title sort is the worst case

If the strongest chunk's document happens to start with 'A', it ends up at the head, fine, by accident. If the strongest chunk's document starts with 'M', it ends up in the dead middle. The placement is uncorrelated with relevance, which is the textbook worst case for any positional bias.

The fix

Order by relevance score. The reranker (Cohere Rerank 3.5, BGE Reranker v2, Voyage Rerank-2) returns a score per candidate; sort descending and assemble in that order. If you want to exploit position bias more aggressively, head-load the top chunk and tail-load the second-best, a head and tail pattern that puts the two strongest chunks into the two highest-attention positions.

Error two, no per-chunk identifier

What a correct assembly looks like

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Anthropic's 2026 RAG cookbook for Claude Opus 4.7 assembles evidence blocks with [S1]-style tags and orders by Cohere Rerank 3.5 score.
OpenAI's Responses API for GPT-5 ships first-class document-id citations, the SDK builds the assembly with stable ids by default.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide between head-loading the top chunk versus head and tail loading?

Head-only is enough when the top chunk dominates relevance; head and tail helps when the top two are close in score and the model otherwise misses the second one.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Sorting evidence blocks by document title rather than by relevance score. Useful chunks land in lost-in-the-middle positions and the model misses them.

60 second bullets to scan on the way to the call.

State the two errors and one fix for each
Explain why alphabetical sorting interacts badly with lost-in-the-middle

Primary sources. Browse if you want the original framing.