Build Log: A Local RAG Pipeline with Ollama and ChromaDB
Retrieval-augmented generation has a reputation as a cloud-only affair, but it doesn’t have to be. This build log assembles a fully local RAG stack: Ollama serving embeddings and chat, ChromaDB storing and searching vectors, and a couple of short Python scripts tying it together. This guide is assembled from official documentation and community reports — we did not run this build ourselves. Every command, default, and caveat below links to the page it comes from.
What You’ll Build (and What “Local” Means Here)
A retrieval-augmented generation pipeline where every component runs on your machine: Ollama serves an embedding model and a chat model, ChromaDB stores and searches document vectors, and a small Python script ties them together. Once the models are pulled, nothing leaves your machine — no API keys and no cloud services.
Picture two pipelines. Ingestion: documents are split into chunks; each chunk becomes a vector — an embedding, a long list of numbers representing its meaning — via Ollama’s embedding model; vectors, text, and metadata land in a Chroma collection. Query: a question is embedded the same way, Chroma returns the closest chunks, and those chunks are pasted into a prompt the chat model answers from. Retrieval supplies the facts; generation writes the sentence.
If you’d rather start with the concepts — what RAG is, why embeddings matter — read our beginner’s guide to local RAG pipelines first. This post assumes that vocabulary and makes the concrete decisions instead: which models, which client mode, which endpoints, and where the documented traps sit.
Stack Choices and Why
Ollama because it exposes a single local HTTP API for both embeddings and chat with a one-line model pull; ChromaDB because it’s a pip-installable vector store with sensible defaults — a default embedding function, a default n_results of 10 — and three deployment modes from in-memory up to a local server. Both are open source; Chroma is Apache-2.0.
Ollama gives the stack one local HTTP surface. The same server that answers chat also answers POST /api/embed (the API introduction covers the rest of the surface); models arrive with a one-line ollama pull; and the embeddings docs document batch input — a single string or an array — plus L2-normalized output vectors, a detail that decides the distance metric later.
ChromaDB is the vector store: pip-installable, Apache-2.0 licensed with roughly 29.5k GitHub stars as of a 2026-10-17 page snapshot. Its getting-started guide demonstrates a working upsert-and-query loop in a few lines, and its clients page offers three ways to run, from throwaway in-memory to a local HTTP server.
One deliberate departure: Chroma ships a default embedding function — Sentence Transformers’ all-MiniLM-L6-v2, downloaded on first use. This build bypasses it and routes embeddings through Ollama so one server powers the whole stack. The Chroma cookbook endorses the pairing and recommends specialized embedding models like nomic-embed-text over general LLMs; the full tradeoff is weighed below.
Prerequisites
You need Python 3 with pip, the Ollama runtime, and disk space: the nomic-embed-text model page lists its tags at 274MB, plus room for a chat model of your choosing. The same page requires Ollama 0.1.26 or later for this embedding model.
- Python 3 and pip — any recent Python 3; pip handles the rest.
- The Ollama runtime, installed for your operating system.
- Disk space — 274MB for the embedding model, plus whatever your chosen chat model weighs; check its model page before pulling.
- Ollama 0.1.26 or later — the model page’s stated floor for nomic-embed-text.
Shopping for local model runners? Our tools directory profiles Ollama and its alternatives.
Step 1 — Install Ollama and Pull the Embedding Model
Install Ollama for your operating system, then pull an embedding model with one command: ollama pull nomic-embed-text. The nomic-embed-text tags (:latest and :v1.5) are 274MB with a 2K context window per the model page, and the model is embedding-only — you’ll pull a separate chat model for generation.
Install Ollama from its official site, open a terminal, and:
# The embedding model this pipeline is built around
ollama pull nomic-embed-text
# Any chat model you like for the generation step —
# pick one from Ollama's library and check its page for size first
ollama pull <your-chat-model>
# Confirm both are installed
ollama list
Both nomic-embed-text:latest and :v1.5 weigh 274MB with a 2K context window, and the page notes the model is embedding-only — no chat, hence the second pull. The same page credits nomic-embed-text-v1.5 on HuggingFace as the upstream model.
Two sanity checks: ollama list from the block above shows what’s installed, and the API reference documents POST /api/ps, which lists currently loaded models. You can even smoke-test embeddings without Python — the Ollama embeddings docs demonstrate a direct CLI call: ollama run embeddinggemma "Hello world".
Step 2 — Install ChromaDB and Pick a Client Mode
pip install chromadb gets you the full package (Chroma’s getting-started guide). Then choose a client mode: chromadb.Client() is in-memory and loses data when the program exits; PersistentClient writes to disk; or run a server with chroma run --path ./chroma_data and connect over HTTP at the documented default, localhost:8000. For a RAG index you’ll keep, pick persistence or server mode.
pip install chromadb
import chromadb
# Option 1 — in-memory: zero setup, but the docs warn that data is
# lost when the program terminates. Fine for throwaway experiments.
client = chromadb.Client()
# Option 2 — persistent: writes the index to a local directory.
client = chromadb.PersistentClient(path="./chroma_data")
# Option 3 — server: run `chroma run --path ./chroma_data` in a
# terminal, then connect a client over HTTP to the documented
# default target, localhost:8000.
For an index you’ll keep, pick option 2 or 3. The getting-started guide demonstrates Client(), collections, upsert, and query in their simplest form; the clients page covers PersistentClient and connecting to a server at localhost:8000.
Step 3 — Ingestion Pipeline: Chunk, Embed, Upsert
Split documents into chunks, send each chunk’s text to Ollama’s POST /api/embed endpoint (which accepts a single string or an array for batching), and upsert the returned vectors into a Chroma collection with metadata (source file, chunk index). Always use the same embedding model for indexing and querying — the Ollama docs call this out explicitly.
Chunking is exactly what it sounds like: cutting documents into pieces small enough to embed and retrieve meaningfully — a few paragraphs, a section, a slide. How big should chunks be? No official page in this log’s sources prescribes a size; it’s a community-tuning question that depends on your documents, and this log won’t invent a number. What the docs do prescribe: the same embedding model at indexing and query time.
import chromadb
import requests
OLLAMA = "http://localhost:11434" # Ollama's local API address
MODEL = "nomic-embed-text" # one model string, reused everywhere
# Persistent client: the index survives restarts (unlike Client())
client = chromadb.PersistentClient(path="./chroma_data")
# Stand-in chunks — in real use, split your own files here
chunks = [
"Ollama serves local models behind a small HTTP API.",
"ChromaDB stores vectors and finds the nearest ones fast.",
"Retrieval-augmented generation grounds answers in your own documents.",
]
# Embed ALL chunks in one request: `input` accepts a string OR an array.
# Returned vectors are L2-normalized (unit length) per the Ollama docs,
# which is why cosine similarity is the recommended metric.
resp = requests.post(
f"{OLLAMA}/api/embed",
json={"model": MODEL, "input": chunks},
)
resp.raise_for_status()
embeddings = resp.json()["embeddings"]
# get_or_create_collection = fetch it if it exists, create it if not.
# "hnsw:space" selects the distance metric (not covered by the sources behind this log; verify in Chroma's collection docs);
# cosine pairs with the normalized vectors, per the Ollama docs' cosine tip.
collection = client.get_or_create_collection(
name="docs",
metadata={"hnsw:space": "cosine"},
)
# upsert = update-or-insert: known ids are overwritten, new ones created.
collection.upsert(
ids=[f"chunk-{i}" for i in range(len(chunks))],
documents=chunks, # kept so answers can show their sources
metadatas=[{"source": "demo", "chunk": i} for i in range(len(chunks))],
embeddings=embeddings, # the vectors from /api/embed
)
print(f"Indexed {len(chunks)} chunks.")
Notice what’s not in that script: an embedding function attached to the collection. Left unattached, a collection falls back to Chroma’s default all-MiniLM-L6-v2 function — fine until you query with text, at which point your question gets embedded by a different model than your chunks. That’s the second failure mode below. The documented fix is attaching an embedding function, which Chroma then uses on add, update, upsert, and query:
from chromadb import EmbeddingFunction # exact import paths in the linked docs
from chromadb.utils.embedding_functions import register_embedding_function
@register_embedding_function
class OllamaEmbeddings(EmbeddingFunction):
"""Lets Chroma call Ollama whenever it needs a vector."""
def __call__(self, input):
resp = requests.post(
f"{OLLAMA}/api/embed",
json={"model": MODEL, "input": list(input)},
)
resp.raise_for_status()
return resp.json()["embeddings"]
@staticmethod
def name(): # stored in the collection's saved config
return "ollama-nomic-embed-text"
def get_config(self): # the settings Chroma saves and replays
return {"model": MODEL}
@staticmethod
def build_from_config(config): # rebuilds the function from that config
return OllamaEmbeddings()
# Attach it at collection creation and Chroma embeds on add, update,
# upsert AND query — no manual requests.post needed:
collection = client.get_or_create_collection(
name="docs",
embedding_function=OllamaEmbeddings(),
metadata={"hnsw:space": "cosine"},
)
That’s the documented extension path: subclass, implement __call__, name, get_config, and build_from_config, register, attach. If you’d rather not maintain a class, Chroma ships an Ollama wrapper in chromadb.utils.embedding_functions per the cookbook, and there’s an npm package, @chroma-core/ollama, for JavaScript (embedding-functions docs).
Step 4 — Query Pipeline: Retrieve, Prompt, Generate
Embed the user’s question with the same model used at indexing time, call collection.query(query_texts=..., n_results=3) to pull the top chunks (Chroma defaults to n_results=10 if you omit it), assemble a prompt from those chunks plus the question, and send it to Ollama’s POST /api/generate with stream:false for a single JSON response. Keep-alive defaults to 5 minutes.
This snippet assumes the collection was created with the Ollama embedding function attached (previous section). Went fully manual instead? Embed the question with the same model and supply vectors on both sides of the search — the Ollama docs’ rule holds either way.
CHAT_MODEL = "<the-chat-model-you-pulled>" # from Step 1
question = "How do I keep my index between restarts?"
# 1) Retrieve. query_texts gets embedded by the collection's attached
# Ollama function — the SAME model as at indexing time. Omit
# n_results and Chroma's documented default kicks in; this asks for 3 (the default is 10).
results = collection.query(query_texts=[question], n_results=3)
# 2) Assemble. Query results come back as lists-per-query, so [0] is
# our single question's hits; stitch them into one context block.
context = "\n---\n".join(results["documents"][0])
prompt = f"""Answer the question using only the context below.
If the context doesn't contain the answer, say so.
Context:
{context}
Question: {question}
"""
# 3) Generate. stream: False returns a single JSON object instead of a
# token stream; keep_alive has a documented default (see table below).
resp = requests.post(
f"{OLLAMA}/api/generate",
json={"model": CHAT_MODEL, "prompt": prompt, "stream": False},
)
resp.raise_for_status()
print(resp.json()["response"])
In plain terms: a chat model only knows what’s in its prompt, so the retrieved chunks get pasted in as context — that’s the “augmented” in retrieval-augmented generation. The template tells the model to answer from that context and admit when it can’t. n_results=3 is the “top-k” idea in one parameter: the three nearest chunks.
One trap from the API’s history: older tutorials use the singular /api/embeddings endpoint with a prompt field. The Ollama API reference marks it superseded by /api/embed and its input field — if a copied snippet sends prompt where you expect input, it’s the deprecated path.
Configuration Table
The table below collects every knob this build touches, its documented default, and the page that documents it: Ollama’s keep_alive of 5 minutes, Chroma’s default n_results of 10, the default HTTP target of localhost:8000, and the default embedding function all-MiniLM-L6-v2 — unused here, but worth knowing exists.
| Setting | This build | Documented default / note | Source |
|---|---|---|---|
| Embed + chat runtime | Ollama — POST /api/embed, POST /api/generate | /api/embed supersedes the singular /api/embeddings | Ollama API reference |
| Embedding model | nomic-embed-text | 274MB tags, 2K context window, requires Ollama 0.1.26+ | model page |
| Embedding dimensions | model-dependent | typically 384–1024 dimensions | Ollama embeddings docs |
| Distance metric | cosine | vectors from /api/embed are L2-normalized; docs tip: use cosine | Ollama embeddings docs |
| Chroma storage mode | PersistentClient(path="./chroma_data") | chromadb.Client() is in-memory; server mode serves over HTTP | Chroma clients |
Top-k (n_results) | 3 | 10 when omitted | Chroma getting-started |
keep_alive | unset | 5 minutes | Ollama API reference |
| Chroma default embedding function | not used | all-MiniLM-L6-v2, downloads on first use | embedding functions |
| HTTP client target (server mode) | localhost:8000 | same — documented default | Chroma clients |
Failure Modes the Documentation Actually Supports
Every failure mode below is backed by a linked official page — no forum folklore. They cover the deprecated embeddings endpoint lurking in older tutorials, mixing embedding models between indexing and query time, silent data loss from the in-memory client, chunks overshooting a 2K context window, and running an Ollama version too old for the embedding model.
- Copying older tutorials with the wrong endpoint. The singular
/api/embeddings, which took apromptfield, is superseded by/api/embedand itsinputfield. Fix: sendmodel+inputto/api/embed. - Different embedding models at index and query time. The Ollama embeddings docs say to use the same model for both — a mismatch puts your question in a different vector space than your chunks. Fix: one
MODELconstant reused everywhere, or an attached embedding function. - In-memory data loss. Chroma’s getting-started docs state it outright: the in-memory client loses data when the program terminates. Fix:
PersistentClient, orchroma run --pathserver mode. - Chunks that outgrow the context window. The model page states a 2K context window for nomic-embed-text; chunk text has to fit inside it. How big is safe? The sources here don’t prescribe sizes — treat it as your tuning problem, not a solved one.
- An Ollama install older than the model requires. nomic-embed-text needs Ollama 0.1.26 or later. Fix: check before pulling — the API reference documents a
/api/versionendpoint that returns the running version.
Deliberately absent: latency figures, accuracy scores, and hardware advice. The sources behind this log don’t quantify those, so neither does the log.
Tradeoffs
Three decisions shape this build: which Chroma client mode (in-memory, persistent, or server over HTTP); which embedding model (specialized nomic-embed-text rather than a general LLM, or Chroma’s default function, which downloads on first use); and context limits — 2K for nomic-embed-text on the chunk side, your chat model’s window on the prompt side.
Client mode. In-memory is zero-setup and ephemeral — data gone when the process ends. PersistentClient trades a directory on disk for durability across restarts. Server mode adds an HTTP hop (default target localhost:8000) but lets several scripts share one index. Start persistent; move to server when a second script needs the same collection. Still choosing a store? Compare vector databases for local AI projects.
Embedding model. Three effort levels: Chroma’s built-in default (zero code, downloads on first use), Chroma’s ready-made Ollama wrapper (cookbook), or your own registered function (embedding-functions docs). The cookbook recommends specialized models like nomic-embed-text over general LLMs — the lane this build stays in. What no source here provides: quality rankings between those models, so none are quoted.
Context windows. nomic-embed-text’s 2K window caps how much text one chunk can carry into the embedding step; your chat model’s own window caps how much retrieved context fits into the prompt. Both ends are bounded — size chunks with the first in mind and keep assembled prompts within the second.
What to Try Next
Three documented next steps: swap the in-memory client for chroma run --path server mode over HTTP; register a custom EmbeddingFunction so Chroma calls Ollama for you on add and query; and batch ingestion through /api/embed’s array input instead of one request per chunk. Each is covered in the official docs linked below.
- Stand up server mode.
chroma run --path ./chroma_data, then connect a client over HTTP so multiple scripts share one persistent index. - Register your embedding function. With
@register_embedding_function, Chroma can rebuild your Ollama-backed function from the collection’s saved config. - Batch the ingestion.
/api/embed’sinputfield accepts an array — one request for the whole document set instead of one per chunk. - Or skip the custom class. Chroma ships an Ollama wrapper in
chromadb.utils.embedding_functions(cookbook); JavaScript gets the npm package@chroma-core/ollama(embedding-functions docs).
FAQ
Short answers to the three questions beginners hit most often. Each traces to the official pages linked in this build log.
Do I need an API key? No. Ollama runs on your machine and serves its HTTP API locally, and Chroma’s client talks to your own process or your own local server — there’s no cloud service in this stack, so there’s nothing to authenticate against. The only “credentials” are model names inside request bodies.
Why did my data disappear?
Because chromadb.Client() is in-memory: Chroma’s getting-started docs state that the client loses data when the program terminates. Anything upserted without persistence lives only as long as the process. Switch to chromadb.PersistentClient(path=...) for disk storage, or start a server with chroma run --path and connect over HTTP.
Can I use my chat model for embeddings too?
The /api/embed endpoint takes whatever model name you send it, but the Chroma cookbook recommends specialized embedding models like nomic-embed-text over general LLMs for embeddings — which is why this build pulls a dedicated embedding model rather than reusing the chat model for double duty.
Sources
Every factual claim above traces to one of these official pages — Ollama’s embeddings docs and API reference, the nomic-embed-text model page, and Chroma’s getting-started, clients, embedding-functions, and cookbook pages. We did not run this build; consult these pages for current behavior.
- Ollama embeddings docs
- Ollama API reference
- Ollama GitHub repository
- nomic-embed-text model page
- nomic-embed-text-v1.5 on HuggingFace
- Chroma getting-started
- Chroma clients
- Chroma embedding functions
- Chroma cookbook — Ollama embeddings
- Chroma on GitHub
📊 See how Ollama and ChromaDB compare →
| Criterion | Score |
|---|---|
| Accuracy | 9/10 |
| Build Detail | 9/10 |
| Beginner Clarity | 9/10 |
| Sourcing | 10/10 |
| Honesty | 10/10 |
| Overall Score | 9.4/10 |
📖 Related Reads
- ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
Cross-links automatically generated from None.
Back to all posts