How to Use Local Embeddings for Private RAG Without Sending Documents to the Cloud

You do not need a cloud API to build retrieval-augmented generation. The embedding model that converts your documents into vectors, the index that stores those vectors, and the language model that writes the final answer can all run on your own hardware. After the initial model downloads, a correctly configured local-only pipeline makes no outbound network calls, which makes it suitable for air-gapped environments and regulated industries.

This guide is written for technically curious beginners and intermediate practitioners who want private question-answering over their own documents. It explains what the embedding step actually does, which components must be replaced to go local, how a documented local-only stack is assembled, and where the approach falls short compared with hosted embedding services.

What the embedding step actually does

An embedding model reads a piece of text and returns a fixed-length list of numbers, called a vector, that represents its meaning. Two passages that discuss the same idea end up with vectors that sit close together in that numeric space, even when they share few exact words. Retrieval works by embedding the user’s question the same way and then asking the vector index for the stored chunks whose vectors are nearest.

A RAG pipeline has three moving parts that talk to AI models: the embedding step that turns documents into vectors, the vector store that holds those vectors, and the synthesis step where a language model reads the retrieved chunks and writes an answer. Most tutorials assume the embedding model and the generator live behind a paid API. They do not have to, and each of those calls can be replaced with a local equivalent.

The embedding step is the one people overlook when they think about privacy. Even if the final answer is generated locally, sending document chunks to a hosted embedding endpoint still transmits your text to a third party. If the goal is that data cannot leave the building, the embedder has to be local too.

Why move the embedding step off the cloud

Three motivations tend to bring people to local embeddings, and they usually arrive in this order.

Privacy and compliance. Teams in healthcare, legal, and finance often cannot paste client data into a third-party API regardless of the vendor’s security certifications. The simplest way to satisfy a privacy review is to make the data physically incapable of leaving the building.

Cost at volume. Embedding large corpora through a paid API adds up quickly. Running the same workload on a local GPU costs electricity, and if you reprocess a large document set repeatedly, the economics shift toward local within weeks.

Control. Local models do not change under you, do not require an API key, and do not impose rate limits or usage dashboards. You choose the model version, the chunking strategy, and the upgrade moment.

These are the arguments made by practitioners who have built private RAG stacks; they describe tradeoffs rather than universal guarantees. Local processing removes the network dependency, but it also moves model selection, evaluation, and maintenance onto you.

A documented local-only component stack

The open-source verifiable-rag project publishes a configuration that runs entirely on one machine, with no API keys and no outbound calls after the initial model downloads. Its component table is a useful reference even if you build your own pipeline, because it names the specific job each local model performs.

Two design details matter here. First, the index layer combines a vector index with BM25, a classic keyword ranking method, so retrieval is hybrid rather than purely semantic. Second, the generator is paired with a separate verifier model whose job is to check whether the generated claims are supported by the retrieved text. That verification stage is what makes the pipeline “verifiable” rather than merely local.

Setting up a local-only pipeline

Step 1: Install only the local extras

The project separates local dependencies from hosted-API SDKs, so you can avoid pulling in client libraries for services you will not call.

pip install "verifiable-rag[pymupdf,bge,lancedb,bm25,litellm,hhem,minicheck]"

This skips the hosted-API SDKs for Cohere and Voyage and is about 50 MB lighter than a full install. The practical benefit is not disk space; it is that the code path for sending text to a hosted embedder is not installed at all.

Step 2: Serve a local language model with Ollama

Ollama is the simplest way to serve a local LLM behind an OpenAI-compatible API, and LiteLLM has built-in support for it. Install Ollama for your platform, then pull a model and start the server:

ollama pull llama3.1
ollama serve

By default Ollama listens on http://localhost:11434. The expected observable result is a running server that answers on that local port; if the port is not listening, the generator step will fail with a connection error rather than a model error.

Step 3: Wire the pipeline

The documented pipeline imports look like this:

from verifiable_rag import Pipeline
from verifiable_rag.chunkers import ParentChildChunker
from verifiable_rag.embedders import SentenceTransformerEmbedder
from verifiable_rag.generators import PromptedCitedGenerator
from verifiable_rag.indexers import BM25Index, HybridIndex, LanceDBIndex
from verifiable_rag.parsers import PyMuPDFParser
from verifiable_rag.rerankers import BGERerankerV2
from verifiable_rag.verifiers import HHEMVerifier

The wiring order follows the table above: parse each document, split it into parent and child chunks, embed the chunks with the local sentence-transformer model, write the vectors and the keyword index to disk, retrieve candidates for a query, optionally rerank them, generate a cited answer with the Ollama-backed model, and verify the claims against the retrieved passages. Constructor arguments and option names are library-version specific, so check the current documentation for your installed release rather than copying a script from an older tutorial.

Step 4: Verify that nothing leaves the machine

A successful local-only run produces an answer with citations to the retrieved chunks, and the only network activity is the initial model download. Confirm this rather than assuming it: watch active connections while a query runs, and check that no hosted-API keys are present in your environment. The first run will download the embedder, reranker, and verifier weights, so a genuinely air-gapped machine needs those files staged beforehand.

Hardware and sizing expectations

The local models in this stack are small by modern standards. BGE-small is roughly 140 MB, the reranker is about 568 MB, and the verifier options are around 600 MB and 770 MB. The embedding model runs on CPU, Apple MPS, or CUDA, so a GPU is helpful but not mandatory for the retrieval half of the pipeline.

Do not treat those file sizes as total memory use. Weight size is only one term; the runtime also needs room for the loaded framework, the working tensors, and, for the generator, the context window and key-value cache. A long context on a large local LLM will dominate memory far more than the 140 MB embedder will. If you are memory constrained, the first thing to cut is the optional reranker, then the second verifier model.

Limitations and tradeoffs worth knowing

Retrieval quality is a smaller-model problem. A compact local embedding model may retrieve worse than a large hosted one on some corpora, particularly with jargon, tables, or unusual phrasing. The only reliable way to know is to build a small labeled set of questions and correct source chunks, then measure recall on your own documents.

Language coverage varies by model. BGE-family models are widely used, but multilingual and domain-specific behavior differs between checkpoints. Check the model card for your language before committing to a corpus-wide re-index.

Scanned documents need OCR. PyMuPDFParser is fast and text-only. If your PDFs are scans or image-heavy, you need the OCR-capable parser, and that path is slower and generally wants a GPU.

Licensing is not uniform. Open weights, open source, and permission for commercial use are three different things. Verify the actual license attached to each embedding, reranker, verifier, and generator model you deploy, rather than assuming that a downloadable model is freely usable in a product.

Local generation is the real bottleneck. Local embeddings are cheap and fast; the local LLM writing the answer is what determines latency and hardware cost. If you only need private retrieval and will generate answers elsewhere, that is a different privacy calculation, because the generation step then sees your retrieved text.

Troubleshooting common failures

• Connection refused when generating. Ollama is not running or is bound to a different port. Start ollama serve and confirm the client points at http://localhost:11434.
• Retrieval returns nothing useful. Usually the parser produced empty or garbled text, most often with scanned PDFs. Inspect the extracted text before blaming the embedder.
• First query is very slow. Expected. Weights are being loaded or downloaded on first use; subsequent queries should be faster.
• Out-of-memory during indexing. Reduce batch size, drop the reranker, or switch to a smaller verifier configuration.
• Answers appear without supporting citations. Check whether the verifier is enabled and whether retrieved chunks actually contain the claim. A verifier flags unsupported statements; it does not fix a retrieval failure.

What to do next

Start with the leanest configuration: local embedder, hybrid file-backed index, no reranker, no verifier. Index a small, representative document set and measure whether the right chunks come back for a dozen questions you already know the answers to. Only then add the reranker and verifier back, one at a time, and re-measure. That sequence tells you which component is actually earning its memory footprint on your corpus.

If you try this on your own documents, the community is interested in the specifics that tutorials usually omit: your hardware, the embedding model and version, chunk size, and the queries where retrieval failed. Sharing those details is what makes the next reader’s setup faster.