toolcall() ← all concepts

// concept · reranking

The right chunk, ranked first.

Your RAG already retrieved the answer — but it's sitting ninth in the list, buried under near-misses, so the model never really reads it. The retrieval wasn't broken. The ranking was.

// the problem

Vector search uses a bi-encoder: it turns your query into one vector and each chunk into one vector, then compares them. Fast and indexable — but it has to compress every possible meaning of a chunk into a single vector, so it's lossy. The result is a rough similarity ranking where the genuinely best chunk can land low.

And low is fatal. LLM recall degrades as you stuff the context window — a right-but-low-ranked chunk near the bottom often gets skipped. Same retrieval, wrong order, vague answer.

The fix

A cross-encoder reranker passes the query and a chunk through the transformer together, and outputs one relevance score. Because attention runs across both at once, every query token can attend to every chunk token — it models interactions a single static vector can't. Far more accurate. You run it over your retrieved chunks and reorder them.

# bi-encoder: encode separately, compare vectors (rough, fast) score = sim( vec(query) , vec(chunk) ) # cross-encoder: encode TOGETHER, score jointly (accurate, slow) score = reranker( query , chunk ) # 0..1 relevance

So why not rerank everything?

There's no precomputed index. A cross-encoder has to run a full transformer inference for every query+chunk pair, at query time. That doesn't scale to millions of docs — Pinecone notes that scoring 40M records with a small reranker would take more than 50 hours, versus under 100 ms for vector search. So you use it as a precision pass, not a search engine.

The two-stage pattern

Retrieve many candidates cheaply with the bi-encoder (high recall), rerank a few with the cross-encoder (high precision), keep the top of those for the LLM. It's one bolt-on step over the top-K you already pull.

# stage 1 — retrieve wide and cheap candidates = index.search(query, top_k=100) # stage 2 — rerank, keep the best few ranked = reranker.rerank(query, candidates) context = ranked[:5] answer = llm(query, context)

Managed rerankers (Cohere Rerank, Pinecone Rerank) are one API call; open cross-encoders (sentence-transformers ms-marco-MiniLM, BGE rerankers) run self-hosted on your own GPU. Either way it's the same one extra step.

The caveats

Reranking is cheap to add, but it isn't free and it isn't magic:

It only reorders what retrieval pulled. If the right chunk isn't in your top-K, the reranker can't invent it — bad chunking or weak embeddings stay bad. It also adds latency and a model call per query (that full transformer pass), and the scores are for ordering, not absolute truth: set a cutoff empirically, don't read the numbers as gospel.

Sources: sentence-transformers — Retrieve & Re-Rank · Cohere — Rerank best practices · Pinecone — Rerankers & two-stage retrieval · Pinecone — Introducing reranking · Hugging Face — sentence-transformers

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click