How to Build a RAG Pipeline with LangChain and OpenAI

Retrieval-augmented generation (RAG) with LangChain and OpenAI is a three-stage pipeline: index your documents, retrieve the passages that match a query, and pass those passages to a chat model to generate an answer. LangChain supplies the document loaders, text splitters, embedding wrappers, and vector-store integrations; OpenAI supplies the embedding model and the chat model. This guide walks through a minimal, working pipeline and the checks that tell you whether it is actually retrieving useful context rather than merely running without errors.

It is written for a technically curious beginner or intermediate practitioner who can read Python and has used an API key before. The code follows the documented API in LangChain’s RAG skill reference; it is presented as a reproducible starting point, not as a benchmark result.

The three stages, and what each one is responsible for

LangChain’s own RAG reference describes the pipeline in three phases. Index covers Load → Split → Embed → Store. Retrieve covers Query → Embed → Search → Return docs. Generate covers Docs + Query → LLM → Response.

Each phase maps to a distinct component type. Document loaders ingest data from files, the web, or databases. Text splitters break documents into chunks. Embeddings convert text into vectors. Vector stores hold those vectors and search them. Keeping the responsibilities separate matters because most RAG failures are localized: a bad answer is usually a retrieval problem, and a retrieval problem is usually a chunking or embedding problem, not a model problem.

Prerequisites and environment

You need Python with pip, an OpenAI API key available as an environment variable, and the LangChain packages that provide the components used below. A single install command covers the documented example:

python3 -m pip install -qU langchain langchain-openai langchain-community langchain-text-splitters

Two environment notes are worth stating up front, because they cause more beginner friction than the code itself.

First, LangChain’s package layout has changed over time. The RAG reference imports embeddings and chat models from langchain_openai, documents from langchain_core.documents, and the splitter from langchain_text_splitters. Older tutorials import the same classes from langchain directly. If an import fails, check the installed version’s documentation rather than assuming the concept is wrong.

Second, version pinning is a real hazard. The Elasticsearch Labs RAG notebook installs openai==0.28.1 and immediately surfaces a resolver warning: langserve 0.0.21 requires pydantic<2,>=1, but you have pydantic 2.3.0 which is incompatible. That is a documented example of a dependency conflict, not a hypothetical one, and it is why a fresh virtual environment per project is the practical default.

Step 1: Load documents

LangChain represents a unit of source material as a Document with page_content and a metadata dictionary. The metadata is not decorative: it is what lets you filter retrieval later by source, date, or tenant.

from langchain_core.documents import Document

docs = [
    Document(
        page_content="LangChain is a framework for LLM apps.",
        metadata={"source": "intro.md"},
    ),
    Document(
        page_content="RAG = Retrieval Augmented Generation.",
        metadata={"source": "glossary.md"},
    ),
]

In a real project you would replace this list with a loader that reads your files, web pages, or database rows. The important property is that the output is a list of Document objects, because every later step expects that shape.

Step 2: Split documents into chunks

Embedding models and context windows operate on limited amounts of text, so documents are split before embedding. LangChain’s reference uses RecursiveCharacterTextSplitter with chunk_size=500 and chunk_overlap=50.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
splits = splitter.split_documents(docs)
print(len(splits))

The overlap exists so that a sentence straddling a chunk boundary is not lost from both chunks. The “recursive” part means the splitter tries progressively smaller separators — paragraphs, then sentences, then characters — to keep chunks semantically coherent rather than cutting mid-thought.

There is no universally correct chunk size. Smaller chunks improve retrieval precision but fragment context; larger chunks preserve context but dilute the embedding with unrelated material. Treat 500/50 as a documented starting point and tune it against your own questions.

Step 3: Embed and store

Embeddings convert each chunk into a vector. The documented example uses OpenAI’s text-embedding-3-small:

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import InMemoryVectorStore

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = InMemoryVectorStore.from_documents(splits, embeddings)

The vector store is the piece most beginners pick arbitrarily. LangChain’s reference gives a short decision table:

The practical consequence: an in-memory store is rebuilt from scratch every time the process restarts, and every rebuild re-embeds every chunk at API cost. If your corpus is more than a toy, choose a store that persists to disk before you write much code around it.

For teams already running Elasticsearch, Elasticsearch Labs documents an alternative path using ElasticsearchStore connected to an Elastic Cloud deployment via its Cloud ID, with OpenAIEmbeddings as the embedding function. That route adds hybrid keyword-plus-vector search and operational tooling, at the cost of running and paying for a search cluster.

Step 4: Retrieve relevant passages

Retrieval embeds the user’s query with the same embedding model and searches the store for the nearest chunk vectors.

query = "What does RAG stand for?"
results = vectorstore.similarity_search(query, k=3)

for doc in results:
    print(doc.metadata.get("source"), "-", doc.page_content)

Two constraints are easy to miss. The query must be embedded with the same model used at index time; mixing models produces vectors that are not comparable and the search silently degrades. And k is a real tradeoff: too few results and the answer lacks context, too many and you push irrelevant text into the prompt, which increases cost and can mislead the model.

Step 5: Generate an answer from the retrieved context

The final step assembles the retrieved chunks and the question into a prompt for a chat model. LangChain’s reference imports ChatOpenAI for this role.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)  # use a model your account can access

context = "nn".join(doc.page_content for doc in results)
prompt = f"""Answer the question using only the context below.
If the context does not contain the answer, say you do not know.

Context:
{context}

Question: {query}
"""

print(llm.invoke(prompt).content)

The instruction to admit ignorance when the context is insufficient is the single most valuable line in the prompt. Without it, a chat model will often answer from its own training data, which makes a broken retrieval stage look like a working system.

Verifying that the pipeline actually works

“It printed an answer” is not a verification. Four checks are cheap and catch most problems.

1. Chunk count. Print len(splits). If it equals the number of input documents, your splitter is not splitting — usually because the documents are already shorter than chunk_size.
2. Retrieval relevance. Print the retrieved chunks before generating anything. If the top results do not contain the answer, no prompt engineering will fix the output.
3. Groundedness. Ask a question whose answer is deliberately absent from the corpus. A correct pipeline says it does not know.
4. Metadata traceability. Confirm each retrieved chunk carries a source you can point a user to.

Common errors and how to read them

Authentication failures. An authentication error naming a missing or invalid key means the environment variable is not visible to the process — check that it is exported in the same shell or container that runs the script, not just defined in a .env file you never loaded.

Dependency conflicts. As the Elasticsearch notebook demonstrates, an older openai pin can collide with a newer pydantic required by other LangChain packages. Isolate the project in a virtual environment and pin versions deliberately.

Dimension mismatch. Switching embedding models after indexing leaves the store holding vectors of the wrong size. The fix is to rebuild the index, not to patch the query path.

Irrelevant retrieval. Usually chunking, not the model. Try smaller chunks, more overlap, or metadata filters that restrict the search space before blaming the embedding model.

Limitations and tradeoffs worth deciding on early

RAG does not eliminate hallucination; it gives the model better material and a reason to stay grounded. If retrieval returns the wrong passage, the answer will be confidently wrong.

Cloud embeddings also mean your documents leave your infrastructure. Every chunk is sent to the embedding API at index time, and every query at retrieval time. Teams with data-residency or confidentiality constraints should evaluate local embedding models and a self-hosted vector store instead — the pipeline shape is identical, only the component implementations change. This is the main reason local-AI setups remain relevant for RAG even when a hosted model would be cheaper per token.

Finally, chunking is a heuristic. It has no notion of your document’s meaning, so tables, code blocks, and multi-page arguments frequently survive splitting in a degraded form. Plan to inspect chunks by hand before trusting the index.

Where to go next

The most valuable next step is not a bigger model — it is a retrieval test set. Collect twenty real questions with the passages that should answer them, then measure whether your pipeline retrieves those passages. That single artifact turns chunk size, k, and vector-store choice from guesswork into decisions you can defend.

From there, persistence (FAISS, Chroma, or a managed store), metadata filtering, and hybrid keyword-plus-vector search are natural extensions. If you are running this pipeline and hit a specific failure — a dependency conflict, an empty retrieval result, or an answer that ignores the context — share your environment, package versions, and the exact error or output in the community so others can reproduce it.