← Cleiton FragaBlog
← Voltar para o blog
·9 min de leitura

From Document to Vector: Chunking, Upsert, and Vector Search

Part 2 of a 5-part series building a production-shaped RAG API on FastAPI + MongoDB Atlas.

From Document to Vector: Chunking, Upsert, and Vector Search

In Part 1 we drew the LLM boundary: embed_texts turns text into vectors, chat_completion turns a prompt into an answer. This post is about everything that happens between a raw document and a relevant chunk landing in front of that chat call. Three operations, in order: chunk the text, upsert the chunks-with-embeddings into the database, and search them by vector similarity at query time.

This is the load-bearing core of RAG. If retrieval surfaces the wrong passages, no amount of prompt engineering or model upgrades saves the answer — the model simply never sees the right text. So it pays to understand each step, including where the reference repo's deliberately simple choices would be the first thing to upgrade in production.

Step 1 — Chunking: why you can't just embed the whole document

You chunk for two hard reasons. First, embedding models have an input token limit, and a 40-page PDF won't fit. Second, and more importantly, a single vector is a lossy summary of whatever you embed. Embed an entire document and you get one fuzzy average of every topic it covers; a query about one specific paragraph will match it weakly, drowned out by everything else. Smaller chunks give sharper, more retrievable vectors — each one about something specific.

The repo's chunker is about as simple as it gets:

# app/rag/chunk.py
def simple_chunk(text: str, max_chars: int = 1000, overlap: int = 100) -> List[str]:
    chunks = []
    start = 0
    n = len(text)
    while start < n:
        end = min(start + max_chars, n)
        chunks.append(text[start:end])
        if end == n:
            break
        start = end - overlap   # step back by `overlap` before the next window
        if start < 0:
            start = 0
    return chunks

This is a fixed-size sliding window with overlap: 1000-character windows, each starting 100 characters before the previous one ended. The overlap is doing real work — it's there so that a sentence split across a chunk boundary still appears whole in at least one chunk. Without overlap, a fact straddling the 1000-character mark gets cut in half and may be unretrievable from either side.

Two limitations are worth naming, because they're the seams where retrieval quality leaks:

It counts characters, not tokens. Embedding models think in tokens (~4 characters of English each), so a 1000-character window is roughly 200–250 tokens. That's well within any model's limit, so it's safe here — but if you bump max_chars up aggressively you can blow past a model's token cap and get a truncated, silently-degraded embedding. Token-aware splitting (using the model's tokenizer to count) removes that risk.

It's blind to structure. The window cuts at character 1000 whether that's mid-word, mid-sentence, or mid-table. It has no idea what a paragraph or heading is. That's the difference between this and the strategy most production systems start with.

The chunking landscape (and what to reach for)

Chunking has become its own small research area. The current consensus, roughly:

  • Recursive character splitting is the recommended baseline and the workhorse of most RAG systems. It tries to split on the largest natural boundary first (paragraph breaks), falling back to sentences, then words, then raw characters — so it keeps semantic units intact when it can. A good starting configuration is 256–512 tokens with 10–15% overlap, which benchmarks well across a wide range of documents.
  • Structure-aware splitting (split on Markdown/HTML headings) is often the single biggest, cheapest win when your documents have structure. A section is a natural chunk.
  • Semantic chunking places boundaries where the topic shifts, detected via embedding similarity between sentences. It buys a few points of recall — but at the cost of embedding every sentence just to decide the boundaries, and the marginal gain shrinks once you've added hybrid search and reranking (which is the entire point of Part 3).
  • Parent–child (hierarchical) chunking indexes small chunks for precise retrieval but feeds the larger parent chunk to the LLM for context. Search precision and generation context, decoupled.
  • Late chunking and contextual retrieval are the 2025-era upgrades: the former embeds a long passage whole (with a long-context model) and then splits the token embeddings so each chunk carries cross-chunk context; the latter prepends a short generated summary or heading-path to each chunk before embedding, making chunks self-contained.

Here's the pragmatic ordering, and it's worth internalizing: start with recursive (or token-based) splitting, then invest in retrieval quality — hybrid search and reranking — before you invest in fancier chunking. Anthropic's own published data shows hybrid + contextual retrieval delivers a bigger lift than switching to semantic chunking. The repo's character-window chunker is a perfectly reasonable v1; the upgrade path is recursive/token-aware first, and the retrieval improvements in Part 3 second.

A useful sanity check before any of this: do you even need chunking? For short, single-purpose documents — FAQs, product blurbs, support tickets — chunking can hurt by fragmenting a document that already matched the question. Chunking earns its keep on long, multi-topic content.

Step 2 — Upsert: chunk, embed, and store atomically-ish

Once you have chunks, you embed them and store them next to the document they came from. The repo's upsert_document does this in three moves:

# app/rag/store.py  (abridged)
async def upsert_document(db, title, content, owner_id):
    # 1) create the parent document, capture its _id
    res = await db["documents"].insert_one(
        {"title": title, "content": content, "owner": owner_id}
    )
    document_id = res.inserted_id

    # 2) chunk + embed (one batched embeddings call for all chunks)
    chunks = simple_chunk(content)
    embeddings = embed_texts(chunks)

    # 3) replace any existing chunks for this document
    await db["chunks"].delete_many({"document_id": document_id})
    docs = [
        {"document_id": document_id, "chunk": chunk_text, "embeddings": emb}
        for chunk_text, emb in zip(chunks, embeddings)
    ]
    if docs:
        await db["chunks"].insert_many(docs)
    return {"doc_id": str(document_id), "chunks": len(docs)}

The data model is two collections with a parent/child relationship:

documents:  { _id, title, content, owner }
chunks:     { _id, document_id, chunk, embeddings }

Each chunk row stores the text (chunk) and its vector (embeddings) and a foreign key (document_id) back to the parent. That foreign key is what lets every query scope itself to a single document and enforce per-user isolation — a detail that matters enormously once you have more than one user.

A few honest observations:

The "upsert" is really an insert. Despite the name, this always creates a new documents row — upload the same file twice and you get two parent documents. The delete_many only clears chunks belonging to the freshly created document_id, which is always empty, so it's a no-op here. The repo's own notes call this out: a true upsert would match on something stable like owner + title and replace in place. As written, it's idempotent-by-accident-only. Worth fixing before you let users re-upload.

Embedding is one batched call. embed_texts(chunks) sends every chunk in a single request rather than looping — the efficiency point from Part 1, applied. For a large document that's the difference between one round-trip and dozens.

There's no transaction. The document insert, the delete, and the chunk insert are three separate operations. If the process dies between them, you can end up with a parent document and no chunks (or vice versa). For correctness under failure you'd wrap these in a MongoDB multi-document transaction, or at least make the operation safely retryable.

Step 3 — Vector search: finding the nearest chunks

At query time you embed the user's question with the same model, then ask MongoDB for the chunks whose vectors sit closest to it. That's the $vectorSearch aggregation stage:

# app/rag/store.py  (abridged)
pipeline = [
    {
        "$vectorSearch": {
            "index": "vector_index",
            "path": "embeddings",
            "queryVector": query_embedding,
            "numCandidates": 200,
            "limit": k,
            "filter": {"document_id": document_id},
        }
    },
    {"$project": {"_id": 1, "chunk": 1, "document_id": 1,
                  "score": {"$meta": "vectorSearchScore"}}},
]
cursor = db["chunks"].aggregate(pipeline)
return [d async for d in cursor]

Read each field, because each one is a decision:

  • index names the Atlas vector index to use. You create this index separately (we'll do that properly in Part 4); it must declare the same dimensionality as your embedding model — 3072 for text-embedding-3-large.
  • path is the document field that holds the vector: embeddings.
  • queryVector is the embedded question.
  • limit is how many results you want back — here k.
  • numCandidates is the knob that trades recall for latency. Atlas runs an approximate nearest-neighbor search (an HNSW graph walk); numCandidates is how many approximate matches it gathers internally before narrowing to your limit. Higher means more accurate but slower. The repo asks for 200 candidates to return ~5–30 results, a ~7–40× over-fetch. The rule of thumb is 10–20× your limit — over-requesting is the main lever for recall in approximate search, and 200 is comfortably in that range.
  • filter pre-filters to a single document before the similarity search. This is what enforces "answer only from this document" and, combined with ownership checks, keeps one user's data out of another's results.
  • $project with {"$meta": "vectorSearchScore"} pulls out the similarity score so you can inspect or threshold on relevance. Note it also drops the bulky embeddings field from the output — you don't want to ship 3072 floats per chunk back to the application for no reason.

Approximate vs exact, briefly

$vectorSearch defaults to ANN (approximate nearest neighbor) via HNSW, which is what makes vector search fast at scale — it doesn't compare your query against every stored vector. For a small collection where you want guaranteed-correct results you can request ENN (exact), which scans everything; it's slower but exhaustive. For most RAG corpora, ANN with a healthy numCandidates is the right default.

What this gets you, and what it doesn't

After these three steps you have a working semantic retriever: ask a question in natural language and get back the chunks that mean something similar, even when they share no keywords with the question. That's the headline capability of vector search and it's genuinely powerful — paraphrase, synonyms, and intent all work.

But pure vector search has a specific, predictable weakness, and it's the reason this series doesn't stop here. Embeddings are great at meaning and surprisingly bad at exact tokens: a part number like XR-4471, an acronym, a proper name, a date — the things where the literal string is the answer — can rank poorly, because semantically XR-4471 and XR-4472 look nearly identical to an embedding model. A user who types an exact code and gets the wrong adjacent code back will not be impressed.

The fix is to run a lexical search alongside the vector search and fuse the rankings, then rerank the survivors with a model that reads the query and each chunk together. That two-phase upgrade — and the MongoDB $rankFusion operator that makes it one query — is Part 3.

Coming up in this series

  1. Hybrid search and reranking — combining vector and BM25 with $rankFusion, then a cross-encoder rerank.
  2. MongoDB Atlas Search — building the vector and text indexes this all depends on.
  3. Putting it all together — the full fastapi-basic-rag request lifecycle.

References