How to Use Local Embedding Models for RAG Instead of Cloud Embeddings

Running embeddings locally means your document text is turned into vectors on your own machine instead of being sent to a hosted embedding API. In practice you load a Sentence Transformers model through a local runtime — LlamaIndex, for example, exposes this as HuggingFaceEmbedding — and compute the vectors on your own CPU or GPU. The trade is straightforward: you take on hardware and maintenance in exchange for privacy, offline operation, and predictable cost.

This guide is for technically curious beginners and intermediate practitioners who already know roughly what retrieval-augmented generation (RAG) does and now want to move the retrieval half of it in-house. It covers when local embeddings are worth it, which models to consider, how to assemble a local pipeline, and the problems that most often trip people up.

Where embeddings sit in a RAG pipeline

A typical RAG pipeline has six stages: ingest documents, split them into chunks, embed each chunk, store the vectors, embed the user’s query and retrieve the nearest chunks, then pass those chunks to a language model to generate an answer. Embedding is the stage most naturally moved off a hosted API, because it is a well-defined, stateless transformation: text in, fixed-length vector out.

That separation matters. You can run embeddings locally while still calling a hosted LLM for generation, or you can go fully local. LlamaIndex documents a complete local stack in which the LLM runs through llama.cpp, vLLM, Hugging Face Transformers, or Ollama; embeddings run through HuggingFaceEmbedding; an optional reranker runs through SentenceTransformerRerank; and vectors live in the default SimpleVectorStore persisted to disk or in a self-hosted store such as Chroma, Qdrant, Postgres/pgvector, or Milvus (LlamaIndex, “Using local models”).

When local embeddings are the right call

Four conditions usually justify the switch.

• Data cannot leave the environment. Contracts, medical notes, internal source code, and regulated records are the classic cases. If the text must not transit a third-party API, local embedding removes that exposure for the retrieval layer.
• Volume makes per-call pricing unpredictable. Re-indexing a large corpus is an upfront cost, but every document edit, format change, or chunking experiment triggers another full pass. On your own hardware that pass costs electricity and time rather than metered calls.
• You need to work offline or air-gapped. A local embedding model downloads once and then runs without network access.
• You want reproducibility. A pinned local model version produces the same vectors tomorrow as today, which is harder to guarantee with a hosted endpoint that may change behind the scenes.

Cloud embeddings remain a reasonable choice when the corpus is small, the content is not sensitive, and you would rather not operate anything. The decision is not ideological; it is about which constraint binds hardest in your case.

Choosing a model: quality versus footprint

Mozilla Builders published a shortlist of English-language embedding models suited to local RAG, released as llamafiles. The three headline recommendations, with the figures reported there, are:

Two caveats belong next to that table. First, those numbers are weight sizes at fp32, not total process memory; the runtime, tokenizer, and batch activations add overhead on top. Second, the comparison was published in May 2024 and explicitly covers English-language models, so treat it as a starting point rather than a current leaderboard (Mozilla Builders, “Using llamafiles for Embeddings in Local RAG Applications”).

For a laptop or a single small server, the 1.25 GB, 1,024-dimension model is the pragmatic default. The 7B-class models at roughly 26 GB of fp32 weights need a machine with real headroom and a GPU to be pleasant to work with.

The rest of the local stack

Embeddings are only one component. LlamaIndex lists the alternatives worth knowing before you commit:

• LLM: llama.cpp, vLLM, Hugging Face Transformers, or Ollama. Any server exposing an OpenAI-compatible API also works through llama-index-llms-openai-like.
• Embeddings: HuggingFaceEmbedding, which runs any Sentence Transformers model locally, with optional ONNX or OpenVINO acceleration on CPU.
• Reranker (optional): SentenceTransformerRerank with a cross-encoder such as cross-encoder/ms-marco-MiniLM-L6-v2 or Qwen/Qwen3-Reranker-0.6B.
• Vector store: the default SimpleVectorStore persisted to disk, or a self-hosted integration such as Chroma, Qdrant, Postgres/pgvector, or Milvus.

The reranker is often the highest-value addition. A cross-encoder rescores the top candidates retrieved by vector similarity, which frequently improves answer quality more than swapping to a larger embedding model would — at the cost of extra latency per query.

Building a local embedding pipeline

Prerequisites. Python and pip, enough disk for the model weights plus your index, and enough RAM to hold the model. The steps below are an illustrative skeleton based on the integrations LlamaIndex documents; check the current documentation for exact package names and import paths, which change between releases.

Step 1 — install the integration packages.

pip install llama-index-embeddings-huggingface llama-index-llms-ollama

The first package provides the local embedding model; the second is only needed if you also want a local LLM.

Step 2 — point the embedding model at a local checkpoint.

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings

Settings.embed_model = HuggingFaceEmbedding(
    model_name="mixedbread-ai/mxbai-embed-large-v1"
)

The first run downloads the weights to the local Hugging Face cache. After that, the model loads from disk and no network call is made.

Step 3 — build and persist the index.

documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)
index.storage_context.persist(persist_dir="./storage")

Step 4 — query it and check the retrieved chunks, not just the answer.

from llama_index.core import StorageContext, load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)
query_engine = index.as_query_engine()
print(query_engine.query("What does the policy say about data retention?"))

The observable success condition is that the retrieved source nodes contain the passage a human would have picked. If the generated answer is wrong but the retrieved chunks are right, your problem is the LLM, not the embeddings — and vice versa.

Memory and hardware planning

Do not plan capacity from the weight-size column alone. A 1.25 GB fp32 checkpoint still needs room for the Python runtime, the tokenizer, and the activations created while embedding a batch of chunks. A 26 GB checkpoint will not fit comfortably on a machine that has 32 GB of total RAM once the operating system, the vector store, and the LLM are also resident.

If memory is tight, the two levers are batch size during indexing and model size. Quantizing an embedding model is possible, but vector spaces are sensitive to numerical precision, so validate retrieval quality on a held-out set of queries before trusting a quantized checkpoint in production.

Common problems and how to diagnose them

• Dimension mismatch after switching models. Different models produce different vector lengths — 1,024 versus 4,096 in the table above — and even same-length vectors from different models are not comparable. Changing the embedding model means re-embedding the entire corpus and rebuilding the index. There is no partial migration.
• Missing prefixes or task instructions. Some embedding models expect queries and passages to be formatted differently, or expect an instruction string. Follow the model card rather than assuming a plain string works.
• Slow first query. The initial call includes a model download and load. Subsequent calls should be much faster; if they are not, the model is being reloaded per request.
• Poor recall on CPU-only machines. The usual fixes are a smaller model, ONNX or OpenVINO acceleration as documented by LlamaIndex, and adding a reranker rather than scaling up the embedding model.
• Everything is slow at index time. Reduce batch size, or accept longer indexing runs. Indexing is a batch job; query latency is what users feel.

Licensing: open weights are not the same as commercial use

The three models in the table carry three different licenses: CC-by-NC, MIT, and Apache 2. Those are not interchangeable. CC-by-NC restricts commercial use; MIT and Apache 2 are permissive. If you are embedding documents for a business, read the model card and the license file before you build a pipeline around a checkpoint, and re-check them if you later swap models.

What to do next

Start with the small Apache-2 model on a representative sample of your own documents, build an index, and measure retrieval quality against twenty questions you already know the answers to. If the retrieved chunks are correct, you have a working local embedding layer and can decide separately whether the LLM and the vector store should also move in-house. If retrieval is weak, add a cross-encoder reranker before you consider a larger embedding model — it is usually the cheaper experiment.

Readers working through this are welcome to share their hardware, chosen model, corpus size, and retrieval results in the community so others can compare configurations.