← Cleiton FragaBlog
← Voltar para o blog
·9 min de leitura

One Client, Two Backends

OpenAI Embeddings and Chat (and Swapping in Ollama)

One Client, Two Backends

A Retrieval-Augmented Generation system touches a language model in exactly two places: it turns text into vectors (embeddings), and it turns a prompt plus retrieved context into an answer (chat completion). Everything else — chunking, vector indexes, hybrid search, reranking — exists to make those two calls produce a good answer.

This first post is about that LLM boundary. The interesting design decision in the reference repo is that it never commits to a vendor. It talks to an OpenAI-compatible endpoint, which means the same forty lines of code drive OpenAI's hosted API in production and a local Ollama server on a laptop, with a one-line configuration change and zero code edits. That property is worth understanding deeply, because it is the cheapest insurance you can buy against vendor lock-in and per-token cost.

The entire LLM layer

Here is the whole client. Two functions, one shared OpenAI instance.

# app/llm/openai_client.py
import os
from typing import List
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

OPENAI_BASE_URL = os.getenv("OPENAI_BASE_URL") or None
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

CLIENT = OpenAI(api_key=OPENAI_API_KEY, base_url=OPENAI_BASE_URL)
EMBEDDING_MODEL = os.getenv("OPENAI_EMBEDDING_MODEL", "text-embedding-3-large")
CHAT_MODEL = os.getenv("OPENAI_CHAT_MODEL", "gpt-4o-mini")


def embed_texts(texts: List[str]) -> List[List[float]]:
    resp = CLIENT.embeddings.create(model=EMBEDDING_MODEL, input=texts)
    return [d.embedding for d in resp.data]


def chat_completion(system_prompt: str, user_prompt: str) -> str:
    resp = CLIENT.chat.completions.create(
        model=CHAT_MODEL,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        temperature=0.2,
    )
    return resp.choices[0].message.content or ""

Three things in here carry more weight than they look.

base_url is the seam. The official openai Python SDK lets you override the API host. Point it nowhere and it hits https://api.openai.com/v1. Point it at any server that speaks the same wire protocol and the SDK doesn't know or care. The repo wires OPENAI_BASE_URL straight into the client constructor — that single environment variable is the entire abstraction.

Models are configuration, not code. EMBEDDING_MODEL and CHAT_MODEL come from the environment with defaults. Swapping gpt-4o-mini for a newer model, or text-embedding-3-large for a cheaper embedder, is a .env edit and a restart.

temperature=0.2 is a grounding decision. In RAG you want the model to lean on the retrieved context, not on its own creativity. A low temperature reduces variance and makes the system answer the same question the same way twice — which you need when you're debugging why an answer was wrong.

Embeddings: what text-embedding-3-large actually gives you

embed_texts sends a list of strings and gets back a list of float vectors. With the default model, each vector is 3072 dimensions long. That number is not cosmetic — it is the single most consequential constant in the whole system, because your vector database index has to be created with the exact same dimensionality. Get it wrong and every query fails with a dimension-mismatch error.

OpenAI currently ships two third-generation embedding models. text-embedding-3-large produces 3072-dimensional vectors and leads OpenAI's own benchmarks; text-embedding-3-small produces 1536-dimensional vectors and is roughly several times cheaper per token. As of mid-2026, the large model runs around $0.13 per million input tokens and the small around $0.02 — embedding is cheap, and it is charged only on input since the output is just numbers. (Always confirm current numbers on OpenAI's pricing page.)

Both models were trained with Matryoshka representation learning, which has a genuinely useful practical consequence: you can pass a dimensions parameter to truncate the vector — say to 1024 or 256 — and the shortened embedding keeps most of its meaning. OpenAI's published result is that a text-embedding-3-large vector cut to 256 dimensions still beats the older ada-002 model at its full 1536. So if your vector store has a dimension ceiling, or you want to cut index size and memory, you don't have to switch models:

resp = CLIENT.embeddings.create(
    model="text-embedding-3-large",
    input=texts,
    dimensions=1024,   # Matryoshka truncation; index must match
)

The reference repo doesn't use this parameter — it stores the native 3072-dim vectors — but it's the first lever to reach for when storage or memory becomes the bottleneck. Just remember the iron rule: whatever dimension you embed at, your Atlas vector index must declare the same number.

A small efficiency note: embed_texts passes the entire list of chunks in one API call. The embeddings endpoint accepts a batch of inputs, so embedding 12 chunks is one round trip, not 12. For large documents this matters; batch your inputs.

Chat: the answer-generation call

chat_completion is a deliberately thin wrapper over the Chat Completions endpoint. It takes a system prompt (the instructions and guardrails) and a user prompt (in RAG, this is the retrieved context plus the user's question), and returns the model's text.

A word on the model default. The repo ships gpt-4o-mini, which was the obvious cost-effective choice when the code was written. The OpenAI lineup has since moved on — as of mid-2026 the current generation is the GPT-5.5 flagship with GPT-5.4 / GPT-5.4-mini / GPT-5.4-nano filling the cost-optimized tiers, and OpenAI now steers new integrations toward the Responses API rather than Chat Completions. None of that requires touching this file: because the model is an environment variable, you migrate by editing .env. If you start a greenfield project, prefer the Responses API; if you have a working Chat Completions integration like this one, it continues to function.

The thinness is the point. There is no streaming, no function calling, no retry logic here — just enough to get an answer. That keeps the RAG orchestration (which we'll cover in Part 5) readable. You add sophistication when you need it, not before.

The trick: same code, local models via Ollama

Now the part that makes the abstraction pay off. The docstring in the repo's client spells it out: set OPENAI_BASE_URL to an Ollama server and the whole thing runs locally.

Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1/. The official OpenAI SDK treats it as a drop-in replacement: you change the base URL, pass any non-empty string as the API key (Ollama requires the field but ignores its value), and your client.chat.completions.create(...) and client.embeddings.create(...) calls work unchanged. For the reference repo, going fully local is three lines of .env:

# .env — point the same client at a local Ollama server
OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_API_KEY=ollama            # required by the SDK, ignored by Ollama
OPENAI_EMBEDDING_MODEL=nomic-embed-text
OPENAI_CHAT_MODEL=llama3.2

# pull the models once
ollama pull nomic-embed-text
ollama pull llama3.2

That's it. embed_texts now calls a local embedding model and chat_completion calls a local chat model, with no change to openai_client.py, store.py, or rag.py. The same abstraction also lets you point at other OpenAI-compatible gateways — Azure OpenAI, vLLM, LiteLLM, or hosted aggregators — by changing only the base URL and model names.

Choosing a local embedding model — and the gotcha that will bite you

Here's where local-vs-hosted stops being free. OpenAI's embedding dimensions don't exist in the open-model world. Common Ollama embedders and their native sizes:

ModelNative dimensionsNotes
nomic-embed-text768Large context, strong general-purpose default
mxbai-embed-large1024Claims parity with larger commercial models at BERT-large size
bge-m31024Multilingual, supports dense + sparse
qwen3-embedding1024 / 2560 / 4096Top multilingual scores; supports a user-defined dimensions parameter
all-minilm384Tiny and fast for quick prototypes

Notice what's missing: nothing outputs 3072, and nothing outputs 1536 natively. If you build your Atlas vector index for text-embedding-3-large (3072) and then switch to nomic-embed-text (768), every query breaks until you drop and recreate the index with numDimensions: 768 and re-embed every chunk. Embeddings from different models are not interchangeable — they don't even live in the same vector space, so you can never mix a query embedded by model A against documents embedded by model B. Switching embedding models always means a full re-index.

If you specifically need 1536 to keep an existing OpenAI-shaped schema, qwen3-embedding accepts a dimensions parameter through Ollama's /v1/embeddings endpoint and can be truncated to match. Otherwise, pick the model first and build the index to fit it.

What the compatibility layer does not cover

The Ollama OpenAI shim is a drop-in for the overwhelming majority of chat and embedding use cases, but a few things differ from the real API: logprobs and n are ignored, native tool/function calling works only with models that support it and may return a slightly different shape, and vision input requires a multimodal model with base64 images. For straight RAG — embed text, retrieve, answer — none of these limitations apply.

Hosted or local? A real decision, not a religious one

The reason to build on the OpenAI-compatible interface is that you don't have to decide once and forever. The honest trade-offs:

  • Cost. Hosted APIs bill per token with zero ops. Ollama costs nothing per token but needs always-on hardware (GPU or a capable machine) and someone to run it. For low, bursty traffic, hosted usually wins; for high steady volume, local can break even.
  • Privacy and residency. With Ollama, no text leaves your machine. For regulated data or air-gapped environments, that can be the whole reason to self-host.
  • Latency. Localhost removes network round-trips, which compounds when a single user query triggers several model calls. But a hosted frontier model often produces a better answer per call.
  • Quality. As of 2026, the strongest hosted models still lead the strongest open models on hard reasoning. The good open embedding models, on the other hand, are genuinely competitive for retrieval.

A common production pattern this architecture enables: local embeddings, hosted chat. Embeddings are high-volume, privacy-sensitive, and the open models are good — run them on Ollama. Answer generation is lower-volume and quality-critical — call a hosted frontier model. Because embeddings and chat read separate OPENAI_BASE_URL-style configuration, you could split them across two clients with a few extra lines. The seam is already there.

One caveat worth flagging before Part 2

This client is synchronous, and it's called from FastAPI's async request handlers. A synchronous network call inside an async def blocks the event loop for the duration of the OpenAI round-trip, which quietly serializes requests under load. The repo's own notes flag this as known technical debt. For a tutorial it's fine; for production you'd move to the SDK's async client (AsyncOpenAI) or push the blocking calls to a thread pool. We'll come back to it in Part 5 when we look at the whole request lifecycle.

Coming up in this series

  1. Chunking, upsert, and vector search — turning a document into searchable vectors, and the $vectorSearch pipeline.
  2. Hybrid search and reranking — why pure vector retrieval misses exact terms, and the two-phase fix.
  3. MongoDB Atlas Search — the data layer that makes lexical and vector retrieval live in one database.
  4. Putting it all together — the full fastapi-basic-rag request lifecycle, end to end.

References