After a cross-encoder reranker was added on every query, costs doubled: keep the quality, cut the bill

Short answer·Medium·4.0 · 0·~3 min·Asked atInflection Ai Infosys Qualcomm

Attempt it

Adding a cross-encoder reranker on every query measurably improved answer quality, but it roughly doubled inference cost because the reranker now scores a wide candidate set on each request. How do you preserve most of the quality gain while cutting the cost?

Free · 2 AI evals / day

The first move is to challenge the assumption baked into the problem: **you do not have to rerank every query over the full candidate set.** That assumption is where most of the cost lives. There are four levers, and you usually combine them. First, **rerank selectively**. Many queries have an obvious top result from the cheap first-stage retriever. Gate the reranker so it only fires when the first-stage scores are close or ambiguous, and skip it when the ranking is already confident. Second, **shrink the candidate count**. A cross-encoder's cost scales with how many candidates it scores. Reranking 25 candidates instead of 100 cuts the bill roughly fourfold, as long as the answer chunk still tends to fall inside the smaller set. Third, **use a cheaper reranker**. A smaller or distilled cross-encoder often recovers most of the quality at a fraction of the cost. Fourth, cache. Repeated and near-duplicate queries can reuse a previous rerank result. Through all of this, measure each change against the quality metric that justified the reranker, so you trade cost for a known, small quality drop instead of regressing silently.

The framing mistake is treating 'rerank every query at full width' as a constant. It is a knob, and most of the doubled cost is being spent on queries that did not need reranking or on a candidate pool wider than necessary. There are four levers, typically combined. Selective reranking gates the cross-encoder on first-stage signals: if the top retrieval scores are well-separated and confident, skip the reranker; fire it only when the ordering is ambiguous. This alone can cut a large fraction of rerank calls because many queries have an unambiguous best chunk. Shrinking the candidate N attacks per-call cost directly. A cross-encoder runs one forward pass per query-candidate pair, so cost is roughly linear in N. Reranking 25 instead of 100 quarters that cost. The right N is set against a recall target on a labeled set, not guessed. Pick the smallest N where the answer chunk still lands in the pool reliably. A cheaper reranker — a distilled or smaller cross-encoder, or a lighter late-interaction model — often recovers most of the quality gain at a fraction of the cost. Caching rerank results for repeated or near-duplicate queries removes redundant scoring. The discipline that ties it together is measurement. Each lever is validated against the same quality metric that justified adding the reranker — answer correctness, NDCG, or faithfulness. That way you trade cost for a known, small quality delta rather than silently regressing. Rerank-everything was never the only option; it was the unexamined default.

TL;DR

Don't rerank every query at full width. Gate the reranker on first-stage ambiguity, shrink the candidate N, use a cheaper distilled reranker, and cache repeats — measuring each against the quality metric.

Memory aid

Easy to grasp

Imagine a busy clinic where every patient, even one with a paper cut, gets sent for a full body scan. The scans help the hard cases, but the bill explodes. A smarter clinic asks a quick question first: does this person actually need the scan? It scans only the unclear cases, uses a faster machine for routine ones, scans a smaller area when the problem is obvious, and reuses yesterday's scan if the same patient returns with the same issue. A reranker is that body scan. You keep its benefit for the cases that need it and stop paying for it on the easy ones, so the bill drops without losing the quality where it mattered.

Key concepts

RAG LLM System Design

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

A reranker is one of the highest-leverage quality upgrades in RAG, and also one of the easiest to deploy carelessly. The default integration — run a cross-encoder over the full first-stage candidate set on every single query — is what most tutorials show, and it is what doubled this team's bill. The quality gain is real, so ripping the reranker out is the wrong response. The right response is to recognize that the reranker's workload is a set of independent knobs, most of which were left at their most expensive setting.

This question separates engineers who treat a reranker as a black box from those who understand its cost model. This deep dive builds that cost model, then works through the four levers — selective firing, candidate width, model capacity, and caching — and closes with the measurement discipline that keeps cost cuts from quietly becoming quality regressions.

The cross-encoder cost model

You cannot optimize what you have not modeled, so start with where a reranker's cost comes from. A cross-encoder scores relevance by feeding the query and a candidate document together through a transformer and reading out a relevance score. The critical word is together: unlike the first-stage bi-encoder, which embeds query and documents separately and compares cheap vectors, the cross-encoder runs a fresh joint forward pass for every query-candidate pair.

That gives a simple cost expression. Per query, the reranker cost is roughly the number of candidates scored times the per-pair forward-pass cost:

\text{cost} \approx Q \cdot N \cdot c_{\text{pair}}

where Q is the number of queries that get reranked, N is the candidate count per query, and c_pair is the cost of one query-candidate forward pass, which scales with the model size.

This expression is the whole optimization map. There are exactly three multiplicative terms, and each one is a lever. Reduce Q by reranking fewer queries. Reduce N by scoring fewer candidates per query. Reduce c_pair by using a smaller model. Caching is a fourth lever that removes work entirely for repeats. The doubled bill came from leaving Q at 100% of traffic, N at the full first-stage width, and c_pair at a large model — all three terms at maximum at once.

Lever one: rerank fewer queries

Levers two and three: narrower candidate set and lighter model

Lever four and the measurement loop that protects quality

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

def retrieve(query, k=5):
    cands = vector_search(query, n=50)
    margin = cands[0].score - cands[1].score
    # Only pay for the cross-encoder when the head is ambiguous
    if margin < MARGIN:
        cands = cross_encoder.rerank(query, cands)
    return cands[:k]

Real products, models, and research that use this idea.

Cohere Rerank is commonly gated to fire only when first-stage retrieval scores are close, skipping confident queries.
Teams swap a large cross-encoder for a distilled MiniLM-based reranker to cut latency while holding most of the NDCG gain.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat concrete first-stage signal would you gate the reranker on, and how would you set the threshold?

Use the score margin between the top candidates from the first-stage retriever — a large gap means the top result is unambiguous and reranking is unlikely to change the order. Set the threshold by sweeping it on a labeled set: plot quality and rerank-call rate against the margin cutoff, and pick the point where skipping more queries starts to cost quality. A small classifier on query features is an alternative gate.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating rerank every query at full width as a fixed constant rather than a knob, so the only optimization considered is a faster model instead of reranking less.

60 second bullets to scan on the way to the call.

Challenge the rerank every query at full width premise as the first move
Explain selective reranking gated on first-stage score ambiguity

Primary sources. Browse if you want the original framing.