Reranker emits its own span with model id, variant tag, input candidate count, output top_n, and per-chunk scores in input order. Offline analysis groups by variant and joins to downstream signals.
Imagine running two coaches on the same youth team in alternating weeks. To know which coach is better, you need to write the coach's name on every practice plan and keep the scores of every drill that practice. At the end of the season you can sort drills by coach name and see which coach's practices produced the better drill scores. If you only kept the scores without writing down the coach, you would see numbers but not know who produced them. The reranker variant tag is the coach name. The per-chunk scores are the drill scores. Without both on the same practice plan, the A/B is unanswerable.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Rerankers are one of the highest-leverage components in a RAG pipeline. A bad reranker undoes good retrieval; a good reranker rescues mediocre retrieval. The cost-benefit of any specific reranker model is empirical, which means you will A/B test rerankers, and you will want the answer to come from your traces rather than from a side by side script.
This deep dive walks through the span structure, the attributes that make A/B analysis tractable, the separation between the feature flag system and the reranker SDK, and the analytical queries the instrumentation enables on Monday morning.
Why the reranker is its own span
A naive instrumentation folds the reranker into the retrieve span. This loses three things.
Latency attribution
The reranker is often slower than the retriever. A hosted cross-encoder might add 80 to 200 ms per call. If reranker time is hidden inside retrieve, latency dashboards cannot answer "is the reranker the bottleneck?" Per-step latency requires per-step spans.
Model-mix slicing
Multiple rerankers may coexist (per tenant, per query type, per experiment). Slicing the model-mix view requires a span per call with the model id as an attribute. Hiding the reranker inside retrieve means the model-mix view shows only the retriever model.
Independent failure modes
Reranker calls can time out, get rate-limited, or fall back. These failure modes deserve their own error attributes and traces. Folded into retrieve, they confuse the retriever's error rate.
The span structure is therefore: retrieve, rerank, generate. Three sibling spans under the trace root, each independently sliceable.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
def rerank_with_tracing(query: str, candidates: list, variant: str, model: str):
with tracer.start_as_current_span('rerank') as span:
span.set_attribute('reranker.model', model)
span.set_attribute('reranker.variant', variant)
span.set_attribute('reranker.input_count', len(candidates))
# Call the reranker; both return per-candidate scores in input order
if model.startswith('cohere'):
results = cohere.rerank(query=query, documents=candidates, model=model)
scores = [r.relevance_score for r in sorted(results, key=lambda x: x.index)]
else:
scores = bge_reranker.score(query, candidates)
# Per-chunk scores in input order (OpenInference convention)
for i, score in enumerate(scores):
span.set_attribute(f'retrieval.documents.{i}.document.score', score)
span.set_attribute(f'retrieval.documents.{i}.document.id', candidates[i].id)
# Output top_n
top_n = sorted(range(len(scores)), key=lambda i: -scores[i])[:5]
span.set_attribute('reranker.output_count', len(top_n))
span.set_attribute('reranker.output_indices', top_n)
return [candidates[i] for i in top_n]Real products, models, and research that use this idea.
- Cohere Rerank API returns a list of {index, relevance_score} for every input document; production stacks emit one score per candidate as openinference.retrieval.documents[].score in input-order.
- BGE-Reranker-v2-M3 is a popular open alternative for self-hosted reranking; the same instrumentation pattern works because the API shape is similar.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect that one reranker is dominating the A/B not because it is better but because it correlates with a routing change upstream?
Check that variant assignment is independent of upstream features (query type, user segment, time of day) by computing chi-square between variant and each feature. If any test rejects independence, the variants are not random; fix the feature flag wiring before trusting any per-arm metric.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Skipping the variant tag, then trying to reconstruct which reranker produced which trace by joining timestamps against deploy logs. Slow, error-prone, often impossible.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.