How to Build a Local RAG Pipeline with Ollama and ChromaDB

A local retrieval-augmented generation (RAG) pipeline does two jobs: it finds the passages in your own documents that relate to a question, then hands those passages to a language model that writes an answer from them. Ollama can serve both the chat model and the embedding model, and ChromaDB stores and searches the resulting vectors. The result runs entirely on your machine — no API keys, no cloud services, and no document text leaving your boundary [1].

This guide is written for a technically curious beginner or intermediate practitioner who is comfortable with a terminal and a little Python. It follows the setup documented in two public walkthroughs: a hands-on tutorial that uses security advisories as the corpus [1], and a LlamaIndex cookbook that runs the whole pipeline locally with Chroma and Ollama [2]. Where the sources do not specify a value, that limitation is stated rather than filled in.

What the pipeline actually does

RAG splits the problem into two phases: retrieval, which finds the right documents, and generation, which answers using those documents as context [1]. In a local build, the flow looks like this:

Documents
   |  chunk + embed
   v
Vector store (ChromaDB)
   ^
   |  similarity search
Query -- embed --> top-k chunks
                      |
                      v
               LLM chat (Ollama)
                      |
                      v
              Grounded answer

The important consequence is that the language model never has to “know” your data. It only has to read the chunks that retrieval placed in front of it. That is what makes a small local model usable for questions about internal advisories, runbooks, or notes it was never trained on.

Two components are easy to confuse. The embedding model converts text into vectors and is used at both indexing and query time. The chat model writes the final answer. In a local Ollama setup these are usually two different models, and mixing them up is one of the most common causes of a pipeline that returns irrelevant context.

Prerequisites and a realistic environment

The cited tutorial lists basic Python knowledge, familiarity with the command line, and a machine with at least 8 GB of RAM as prerequisites [1]. The cookbook adds that Ollama must be installed and running, with two models pulled locally [2].

• Ollama installed and serving requests on your machine.
• A chat model, for example llama3.2:3b.
• An embedding model, for example nomic-embed-text.
• Python with the LlamaIndex and Chroma packages installed.

Note what these sources do and do not specify. They name the models and the memory floor; they do not publish benchmark timings, GPU requirements, or a recommended chunk size. Treat any specific number you see elsewhere as a starting point to measure, not a guarantee.

Step 1 — Install Ollama and pull both models

Install Ollama following its own instructions, confirm it is running, then pull the two models the cookbook uses [2]:

ollama pull llama3.2:3b
ollama pull nomic-embed-text
ollama list

Expected result: ollama list shows both entries. If the embedding model is missing, the indexing step will fail or fall back to something you did not intend, so check this before writing any Python.

Step 2 — Install the Python stack

The cookbook installs the LlamaIndex integration packages plus Chroma [2]:

pip install llama-index-core llama-index-llms-ollama 
  llama-index-embeddings-ollama llama-index-vector-stores-chroma 
  llama-index-readers-file chromadb

Expected result: the imports in the next step resolve without a ModuleNotFoundError. If one package is missing, install it individually rather than reinstalling the whole list.

Step 3 — Ingest and chunk your documents

Chunking breaks long documents into passages small enough to embed and retrieve precisely. The cookbook uses SentenceSplitter for this job [2]. Chunk size is a tradeoff: chunks that are too large dilute the embedding with unrelated text, while chunks that are too small lose the context needed to answer. The cited sources do not prescribe a chunk size, so the value in the script below is illustrative only — change it and observe the effect on retrieval.

Attach metadata such as the source file path to each document. When a retrieved chunk looks wrong, the metadata is what tells you which file produced it.

Step 4 — Embed the chunks and persist them in ChromaDB

Embedding converts each chunk into a vector using OllamaEmbedding, and ChromaDB stores those vectors for later similarity search [2]. Persisting to disk matters: without it, you re-embed the entire corpus on every run, which is the slowest part of the pipeline.

Step 5 — Retrieve and generate an answer

At query time the same embedding model converts the question into a vector, ChromaDB returns the closest chunks, and the chat model answers using those chunks as context. The number of chunks passed forward is the top-k setting. Raising it gives the model more material but also more noise and a longer prompt.

A complete illustrative script

The following sketch combines the documented APIs from the cookbook [2]. It is an illustrative example, not a record of a test run by this publication, and the chunk parameters are starting values rather than recommendations.

import chromadb
from llama_index.core import Document, StorageContext, VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama
from llama_index.vector_stores.chroma import ChromaVectorStore

embed_model = OllamaEmbedding(model_name="nomic-embed-text")
llm = Ollama(model="llama3.2:3b", request_timeout=120.0)

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("advisories")
vector_store = ChromaVectorStore(chroma_collection=collection)

splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
documents = [Document(text=open(p).read(), metadata={"path": p}) for p in paths]

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
    embed_model=embed_model,
    transformations=[splitter],
)

query_engine = index.as_query_engine(llm=llm, similarity_top_k=5)
print(query_engine.query("Which advisories affect the VPN gateway?"))

Verify the pipeline before you trust the answer

A fluent answer is not evidence that retrieval worked. Check the retrieval layer first:

1. Confirm the collection contains the number of chunks you expect for your corpus.
2. Print the retrieved chunks for a question whose answer you already know, and confirm the answer is actually present in them.
3. Ask a question about something deliberately absent from the corpus. If the model still answers confidently, your prompt is not enforcing grounding.

The cookbook treats these as first-class concerns, listing empty context, long queries, and hallucination guards among the failure modes to explore [2]. An empty-context result usually means the embedding model differs between indexing and querying, or that the collection was never populated.

Measuring retrieval quality with hit-rate and MRR

The cookbook evaluates retrieval against a gold Q&A set using hit-rate and mean reciprocal rank (MRR) [2]. Both require you to write down, for a set of questions, which document should be retrieved. Hit-rate tells you how often the correct document appears in the returned results at all; MRR rewards ranking it near the top. Together they separate two different problems: “the right passage was never retrieved” and “it was retrieved but buried.” Fixing chunking and top-k helps the first; reranking helps the second.

Common failure modes and fixes

• Empty or irrelevant context. Check that the same embedding model is used for indexing and querying, and that the collection is not empty.
• Long queries. Very long questions embed poorly. Reduce the query to its core terms, or split it into separate questions.
• Ungrounded answers. Add an explicit instruction to answer only from the provided context and to say when the context is insufficient — the cookbook’s hallucination guard [2].
• Model not found. Re-run ollama list; a missing model name is a configuration error, not a retrieval problem.
• Memory pressure. Both the embedding model and the chat model occupy memory at once. Weight size is not the whole story — context length and runtime overhead add to it.

Privacy, licensing, and operational tradeoffs

The main appeal of this architecture is boundary control. The cited tutorial frames the whole point as running with “no API keys, cloud services, or data leaving the boundary” [1], and the cookbook notes that no API key is required because both the LLM and the embeddings are local [2]. That is a genuine advantage for security advisories, internal notes, or client material you would rather not upload.

Two cautions. First, local does not automatically mean private if you later add a cloud connector or a hosted reranker. Second, freely downloadable weights are not the same as open source or commercially unrestricted use; check the license that ships with each specific model rather than assuming. A small local chat model also tends to be less capable than a large hosted one, which is precisely why retrieval quality matters more here, not less.

Where to go next

Build the smallest useful version first: one folder of documents, one embedding model, one chat model, and a handful of questions whose answers you already know. Once retrieval is measurable, tune chunking and top-k, then decide whether you need reranking or better document parsing. If you hit an error or a surprisingly bad retrieval result, share your environment — operating system, Ollama version, model names, and the exact error text — in the community so others can reproduce it.