← Cleiton FragaBlog
← Voltar para o blog
·10 min de leitura

Two-Phase Retrieval: Hybrid Search with `$rankFusion` and Cross-Encoder Reranking

Part 3 of a 5-part series building a production-shaped RAG API on FastAPI + MongoDB Atlas

Two-Phase Retrieval: Hybrid Search with `$rankFusion` and Cross-Encoder Reranking

Part 2 ended on a confession: pure vector search is excellent at meaning and unreliable at exact tokens. Ask it about a part number, an acronym, a person's name, or a date, and it can confidently return a semantically adjacent chunk — XR-4472 when you asked for XR-4471 — because to an embedding model those strings look almost the same.

This post is the fix, and it's the part of the pipeline where most of the quality gains in a serious RAG system actually live. The pattern is two-phase retrieval: a cheap, wide first pass that maximizes recall (don't let the right chunk escape), followed by an expensive, precise second pass that maximizes precision (make sure the best chunk lands on top). In the reference repo:

embed question
  → ① hybrid search  (vector + BM25, fused) → ~30 candidates   [maximize recall]
  → ② rerank         (cross-encoder)         → top 5            [maximize precision]
  → LLM

Two phases, because a single retriever is forced to choose between recall and precision. Split them and each one optimizes its own metric.

Phase 1 — Hybrid search: meaning and keywords

The insight behind hybrid search is that vector search and lexical search fail in opposite directions, so running both and combining them covers each other's blind spots.

RetrieverMechanismStrong atWeak at
Vector ($vectorSearch)semantic similarity in embedding spaceparaphrase, synonyms, intentexact strings, codes, acronyms, numbers
Lexical ($search, BM25)term-frequency keyword scoringexact tokens, rare terms, identifierssynonyms, rephrasings, intent

BM25 is the classic full-text ranking function — it rewards documents that contain the query's terms, weighting rarer terms more heavily and dampening the effect of very long documents. It will nail XR-4471 precisely because it matches the literal token. Vector search will nail "how do I reset the device" matching a chunk that says "restoring factory defaults" with no shared words. You want both.

Fusing two rankings: Reciprocal Rank Fusion

The hard part of hybrid search isn't running two queries — it's combining two ranked lists whose scores aren't comparable. A cosine similarity of 0.82 and a BM25 score of 14.3 live on completely different scales; you can't just add them. The standard solution is Reciprocal Rank Fusion (RRF), which throws away the raw scores and uses only rank position:

RRF(doc) = Σ  1 / (k + rank_in_list_i)
          over each list i the doc appears in

The constant k (MongoDB uses 60) smooths the curve so that the difference between rank 1 and rank 2 isn't wildly larger than between rank 50 and rank 51. A document that ranks well in both lists accumulates score from both and floats to the top; a document that only one retriever liked still contributes. Because RRF operates purely on ranks, it's robust to the incompatible score scales of dense and sparse retrievers — which is exactly why it's the default fusion method in production search systems.

$rankFusion: one query, two retrievers, native fusion

Historically you implemented this client-side: run the vector query, run the text query, pull both result sets back to the application, merge with RRF in Python. MongoDB now does it inside a single aggregation pipeline with the $rankFusion stage. Here's the repo's hybrid_search_chunks, abridged:

# app/rag/store.py  (abridged)
pipeline = [
    {
        "$rankFusion": {
            "input": {
                "pipelines": {
                    "vector": [
                        {"$vectorSearch": {
                            "index": "vector_index",
                            "path": "embeddings",
                            "queryVector": query_embedding,
                            "numCandidates": 200,
                            "limit": k,
                            "filter": {"document_id": document_id},
                        }}
                    ],
                    "text": [
                        {"$search": {
                            "index": "text_index",
                            "compound": {
                                "must": [
                                    {"text": {"query": query_text, "path": "chunk"}}
                                ],
                                "filter": [
                                    {"equals": {"path": "document_id", "value": document_id}}
                                ],
                            },
                        }},
                        {"$limit": k},
                    ],
                }
            },
            "combination": {"weights": {"vector": vector_weight, "text": text_weight}},
        }
    },
    {"$limit": k},
    {"$project": {"_id": 1, "chunk": 1, "document_id": 1,
                  "score": {"$meta": "score"}}},
]

What's happening:

  • Two named sub-pipelines, vector and text, run independently inside one stage. The vector branch is the $vectorSearch from Part 2; the text branch is an Atlas $search doing BM25 over the chunk field. Both branches filter by document_id, so the per-document, per-user isolation holds across both retrievers.
  • combination.weights lets you bias the fusion. The repo defaults to vector=0.7, text=0.3 — leaning semantic but keeping a meaningful lexical contribution. These are tunable: bump text up for a corpus full of codes and identifiers, bump vector up for prose.
  • $rankFusion applies RRF across the two rankings and writes the combined score into the score metadata field, which the final $project reads via {"$meta": "score"}.
  • The output is ~30 candidates (the candidates request parameter, passed as k here), deliberately wide. This phase is tuned for recall — get the right chunk somewhere in the set — not for final ordering.

Version requirement, stated precisely. The $rankFusion operator requires MongoDB 8.0+. The ability to fuse $search and $vectorSearch pipelines natively under it — which is what this code does — landed in Atlas 8.1+, which is what the repo targets. If you're on an older cluster, the portable fallback is to run the two queries separately and do RRF in application code (two database round-trips instead of one, but works anywhere).

After Phase 1 you have ~30 chunks that are probably relevant, in a roughly right order. Good enough to pass to a model that does ordering properly.

Phase 2 — Reranking: read the query and the chunk together

Here's the subtle reason retrieval needs a second pass. Both $vectorSearch and BM25 score each chunk in isolation from the query. Vector search embeds the query once and the chunk once — two separate forward passes — and compares the results with cosine similarity. That's a bi-encoder: fast (you precompute all the document vectors), but it never lets the query and the chunk interact. It's comparing two summaries.

A cross-encoder does the opposite. It feeds the query and a candidate chunk into the model together, in one forward pass, and outputs a single relevance score that reflects how well this specific chunk answers this specific query. Because the model attends across both texts jointly, it catches relevance that cosine similarity misses — and it's dramatically more accurate at ordering. The catch is cost: you can't precompute anything, so you run one model inference per candidate. That's exactly why you only rerank ~30 survivors, never the whole collection.

This is the classic recall/precision division of labor:

cheap, wide, isolated scoring   →   expensive, narrow, joint scoring
   hybrid search (~30)                    rerank (top 5)
   "don't miss the chunk"                 "put the best chunk first"

The repo's reranker (and its best design choice)

The reference implementation calls a hosted reranking API over plain HTTP — no heavy SDK, no local model:

# app/rag/rerank.py  (abridged)
def rerank(query, candidates, top_n):
    if not candidates:
        return []
    if not is_enabled():                 # no API key configured
        return candidates[:top_n]        # ← graceful degradation

    documents = [c.get("chunk", "") for c in candidates]
    try:
        resp = httpx.post(
            f"{RERANK_BASE_URL}/v2/rerank",
            headers={"Authorization": f"Bearer {RERANK_API_KEY}",
                     "Content-Type": "application/json"},
            json={"model": RERANK_MODEL, "query": query,
                  "documents": documents, "top_n": min(top_n, len(documents))},
            timeout=30.0,
        )
        resp.raise_for_status()
        results = resp.json().get("results", [])
    except Exception as exc:             # network / auth / payload error
        logger.warning("Rerank failed, using original order: %s", exc)
        return candidates[:top_n]        # ← degrade, never break /ask

    reranked = []
    for r in results:
        idx = r.get("index")
        if idx is None or idx >= len(candidates):
            continue
        item = dict(candidates[idx])
        item["rerank_score"] = r.get("relevance_score")
        reranked.append(item)
    return reranked or candidates[:top_n]

The defaults target Cohere Rerank (RERANK_BASE_URL=https://api.cohere.com, model rerank-v3.5), which exposes a /v2/rerank endpoint: send a query and a list of document strings, get back an array of {index, relevance_score} sorted by relevance. The function maps those indices back onto the original candidate objects and tags each with its rerank_score. rerank-v3.5 handles 100+ languages and up to ~4,096 tokens per document, which comfortably covers ~1000-character chunks.

The single best decision in this file is graceful degradation. Reranking is treated as an optional enhancement layer, not a hard dependency. No API key configured? rerank returns the hybrid candidates in their existing order. Network blip, auth failure, malformed response? Same — it logs a warning and falls back. The /ask endpoint never breaks because an external service hiccuped, and even without reranking the hybrid search alone is already a clear improvement over pure vector search. That's the right posture for any third-party call on a request's critical path: enhance when available, degrade quietly when not.

Two parameters that control the whole trade-off

Because the cost and latency of the pipeline concentrate almost entirely in the rerank call, the repo exposes two request parameters that let callers tune it per query:

  • candidates (default 30) — how many chunks Phase 1 retrieves. More candidates means better recall (less chance of missing the right chunk) but more text to rerank. Retrieving 30 instead of 5 from MongoDB is nearly free; the cost shows up downstream.
  • k (default 5) — how many chunks survive reranking and go into the LLM's context. More context can help the answer but costs more chat tokens and risks diluting the prompt with marginal chunks.

The mental model: candidates buys recall, k controls how much context the model reasons over, and the rerank in between is where you pay for precision. Tune candidates up on a hard corpus; keep k tight to keep the prompt focused.

Picking a reranker — hosted or self-hosted

The repo's HTTP shape is intentionally generic. Because it just POSTs to {base_url}/v2/rerank with {model, query, documents, top_n}, you can repoint it at other providers that mirror that contract — Jina and Voyage both offer compatible reranking endpoints — by changing RERANK_BASE_URL and RERANK_MODEL. Your options, roughly:

  • Cohere Rerank (rerank-v3.5, and the newer Rerank 4.0 rerank-v4.0-pro / -fast) — the path of least resistance: a managed endpoint, strong multilingual quality, predictable per-search pricing (around $2 per 1,000 searches, where a search is one query against up to ~100 documents).
  • Self-hosted cross-encoders — the BGE reranker v2 family (bge-reranker-v2-m3 is the multilingual default, Apache-licensed) and Jina Reranker v2 run on your own GPU with no per-call cost. Accuracy is close to Cohere on standard benchmarks; the real cost is the ops work — deployment, monitoring, GPU bills — which is usually what decides build-vs-buy, not the model quality.
  • ColBERT-style late interaction sits in between, offering a different precision/cost profile.

A caveat worth keeping honest: reranking doesn't always help. If your first-stage retrieval is already returning the right chunks in the right order, a reranker adds latency and cost for no gain — and a poorly matched one can even regress quality. The way you find out is the same way you tune any of this: build a small evaluation set of real questions with known-good answers and measure. Without measurement, every knob in this post is a guess.

Where we are

With hybrid search and reranking in place, the retriever is no longer fooled by exact-token queries and no longer dependent on cosine similarity getting the final ordering right. The chunks that reach the LLM are the ones a precise model judged most relevant to this question — and the whole thing degrades gracefully if the rerank service is unavailable.

All of this leans on one substrate: a single MongoDB collection that holds both a vector index and a text index, queried together under $rankFusion. Part 4 builds those indexes and explains the Atlas Search machinery underneath them.

Coming up in this series

  1. MongoDB Atlas Search — the vector and text indexes, the operators, and how they coexist in one database.
  2. Putting it all together — the full fastapi-basic-rag request lifecycle, end to end.

References