How to Build a RAG-Powered Content Assistant That Avoids Repeating Existing WordPress Posts

A RAG-powered content assistant avoids repeating your existing WordPress posts by indexing every published article into a vector store, then querying that store before the model writes anything. Retrieval surfaces the posts that already cover the topic; generation is then instructed to build on them, link to them, or stop and flag the overlap. Retrieval-augmented generation does not do this by itself — the duplicate check is a deliberate step you add to the pipeline.

This guide is for WordPress site owners and developers who publish regularly and want a drafting assistant that knows what is already on the site. It assumes basic familiarity with the WordPress REST API and a scripting language such as TypeScript or Python. Two public project documents serve as reference designs: the F08 RAG knowledge base specification, which defines the vector store and embedding interfaces, and gyomu-system-rag, a production RAG search system built for a Japanese WordPress tech blog.

The two flows you need to build

The gyomu-system-rag project describes its architecture as two distinct flows. The indexing flow turns articles into searchable vectors: a WordPress article is fetched through the REST API, chunked by heading, embedded, and written to the vector store. The query flow runs in reverse — a question or draft topic is embedded, the nearest chunks are retrieved, and an LLM answers only from those passages.

For a duplicate-avoidance assistant, the query flow is the important one, but it is only as good as the indexing flow behind it. If a post was never indexed, or was indexed under an old revision, the assistant will happily rewrite content that already exists.

Prerequisites and component choices

Vector store: sqlite-vec or pgvector

The F08 specification defines a single VectorStore interface with two adapters: sqlite-vec for a local SQLite file, and pgvector for a Postgres connection such as Supabase. The interface is deliberately narrow, which keeps the two adapters swappable:

export interface VectorStore {
  initialize(): Promise<void>;
  upsertDocument(collection: string, slug: string, chunks: VectorChunk[]): Promise<void>;
  removeDocument(collection: string, slug: string): Promise<void>;
  query(embedding: number[], options: { limit?: number; collection?: string }): Promise<VectorChunk[]>;
  close(): Promise<void>;
}

For a single-site WordPress install, sqlite-vec keeps everything on the same machine as the site and avoids a network hop per query. pgvector makes more sense when several editors or services need to query the same index, or when you already run Postgres. The choice affects operations, not the duplicate-check logic.

Embedding model

The F08 design defaults to text-embedding-3-small and exposes the model name as a configuration field. Whichever model you pick, treat it as a schema decision rather than a setting: vectors from two different models are not comparable, so changing the model means re-embedding the entire corpus. Record the model name and version alongside the index so a future migration is a known task rather than a surprise.

Step 1 — Pull published posts through the WordPress REST API

Index only what readers can see. The gyomu-system-rag indexing flow fetches article bodies through the WordPress REST API and re-indexes an article when it is published, updated, or deleted. Drafts, private posts, and revisions should be excluded at the fetch stage, otherwise the assistant will flag overlaps with content that does not exist publicly.

Store the post slug alongside each chunk. The F08 VectorChunk type carries documentSlug, documentTitle, the source field name, and a position index, which is enough to reconstruct a citation back to the live URL.

Step 2 — Chunk by heading, not by character count

Chunking determines what “similar” means. gyomu-system-rag chunks by heading, which keeps each chunk topically coherent and makes retrieved passages readable when they are pasted into a prompt. The F08 specification offers a token-based default instead: chunkSize of 512 tokens with chunkOverlap of 50 tokens.

Both approaches are workable. Heading-based chunking tends to produce better citations because a chunk maps to a section a reader can actually navigate to. Token-based chunking gives more predictable embedding costs and handles long unbroken sections. If you use token chunking, keep the overlap — it prevents a sentence that straddles a boundary from being invisible to retrieval.

Step 3 — Make indexing idempotent

This is the step most homegrown pipelines skip. gyomu-system-rag describes its indexing as idempotent differential: when an article changes, only that article is re-indexed. That property matters for duplicate detection, because a stale index is worse than no index — the assistant will confidently report that nothing similar exists while an updated post sits unread.

The F08 interface supports this directly. upsertDocument replaces all chunks for a given collection and slug, and removeDocument clears them when a post is deleted or unpublished. Wire both to WordPress hooks so the index follows the site rather than a nightly full rebuild.

Step 4 — Query before you generate

The duplicate check is a retrieval call made before any drafting happens. Embed the proposed topic or outline, ask the store for the nearest chunks, and inspect the top results:

const embedding = await embeddings.embed(proposedTopic);
const matches = await store.query(embedding, { limit: 5, collection: 'posts' });

// matches[0] is the closest chunk; its documentSlug and documentTitle
// tell you which published post already covers this ground.

One caveat is worth flagging. The published query() signature returns VectorChunk[] and the documented VectorChunk type has no score field, so a similarity threshold is not available out of the box. You will need to extend the return type to carry the distance or score your store produces, or compute it in the adapter. Without a score you can still rank results, but you cannot say “this is close enough to be a duplicate” versus “this is merely related.”

Set the threshold empirically. Index your own site, run a handful of topics you know are already covered and a handful you know are new, and pick a cutoff that separates them. There is no universal number.

Step 5 — Put the retrieved posts into the generation prompt

Retrieval only prevents repetition if the generator is told what was retrieved. Pass the matched titles, slugs, and passages into the prompt with an explicit instruction set: build on the existing post, link to it, or decline and explain the overlap. gyomu-system-rag takes the same posture in its reader-facing widget — the LLM answers only from retrieved passages and always cites the source articles.

If the assistant is drafting rather than answering, the useful outputs are a short overlap report plus a recommendation: extend the existing post, write a distinct angle, or proceed as new. Human review before publication remains the right default for anything factual.

Step 6 — Verify with a deliberate duplicate test

A reproducible check is more convincing than a demo. Pick a post you published months ago, feed its title to the assistant as a fresh topic, and confirm that the retrieval step returns that post as the top match and that the prompt response flags the overlap. Then pick a genuinely new topic and confirm the assistant does not flag it. If the new topic triggers a false positive, your threshold is too loose; if the old topic is missed, check whether the post was indexed at all before adjusting anything else.

Limitations, costs, and failure modes

RAG reduces repetition; it does not eliminate it. Three constraints are worth planning for.

Public exposure needs guards. gyomu-system-rag lists rate limiting, cost caps, and prompt-injection defense as production safeguards added before public exposure. If your assistant sits behind a chat widget rather than an editor-only tool, the same concerns apply — retrieved post content is untrusted input from the model’s perspective.

Embedding and chunking choices are sticky. Changing the embedding model or the chunk size invalidates the existing index and requires a full re-embed. Budget for that as a maintenance event, not a config tweak.

Similarity is not identity. Two posts can cover the same topic with different framing, and a single post can span several topics. Retrieval will surface both cases; the threshold and the prompt decide what to do about them. Expect to tune both after the first few weeks of real use.

Next step

Start with the indexing flow alone. Fetch published posts, chunk them, embed them, and store them with slugs attached. Once retrieval returns sensible results for a handful of known topics, add the pre-generation duplicate check. That ordering keeps the failure surface small: if the assistant flags the wrong posts, you are debugging retrieval rather than retrieval, prompting, and generation at once.

If you build this and hit a specific problem — an index that will not stay current, a threshold that will not settle, or a chunking scheme that breaks on your content — share your WordPress version, vector store adapter, embedding model, and the exact retrieval output in the Ai-Forum.org community so others can reproduce it.