Bulk AI writing is best handled as a queue, not a loop. A planner creates one durable job per article, a worker pulls jobs at a rate you control, and the finished draft returns through a callback to the system that requested it. That separation is what makes fifty titles survivable: a failure affects one job instead of the whole run, and throughput becomes a setting rather than an accident.
This guide is for practitioners who can already generate a single article with an AI tool or workflow and now need to do it in batches. It covers the two-stage model, a published WordPress-to-Celery reference architecture, the failure modes that matter most (duplicates, retries, rate limits), and where a human review checkpoint belongs.
Why a loop breaks down at scale
The obvious first implementation is a loop: read a list of titles, call the model for each one, write the result into the CMS. It works for three articles and fails predictably at thirty. Long HTTP requests time out mid-generation. One provider error aborts the whole run, and there is no record of which titles completed. Nothing throttles the request rate, so a burst of parallel calls hits provider limits. And there is no visibility: once the script exits, the operator cannot tell whether a title was skipped, retried, or silently half-written.
A queue replaces the loop with durable intermediate state. Each article becomes a job (also called a task) stored in a broker such as Redis. A worker process pulls one job at a time, calls the model, and reports the result back. Because the job exists independently of the process that created it, retries, pacing, and monitoring all become possible. The vocabulary is worth fixing early: enqueue means adding a job, dequeue means a worker claiming it, and a callback or webhook is how the result travels back to the originating site.
Planning and generating are two different stages
The most common conceptual mistake is assuming that planning a batch also generates it. In jekcms, the documented behavior is explicit: Plan Many Posts takes a list of titles, one per line, plus a category and a spread setting, and writes one task per title with its own date and time. Fifty titles at two per day becomes twenty-five days of scheduled work instead of a fifty-post dump. But planning a batch generates nothing. The tasks sit in the queue until a writer picks them up, and that writer is an n8n workflow the operator runs.
The division of labor is deliberate. jekcms hands out the task; your workflow calls the AI provider; the finished article comes back through the API. The stated reason is control: the prompt, the persona, the image generation, and the model are things you should own rather than settings buried in a CMS screen. The same documentation separates this from two adjacent features that are easy to confuse with it. AI Draft generates a single post in the browser from a topic and a writing style and drops it straight into the editor. The Content Queue is a different pipeline for drafts that arrive already written. Both appear on the shared Content Studio calendar, which is why they look similar at a glance.
A reference architecture: WordPress, Django, Celery, and Redis
When the CMS is WordPress and the generation backend is Python, a documented pattern is to sign a job request in WordPress, verify and enqueue it in Django, run the model call in a Celery worker backed by Redis, then post the result back to WordPress. The pipeline has four responsibilities:
- WordPress (PHP): a button or cron trigger sends a signed POST to a
/jobsendpoint. - Django API: verifies the HMAC signature and enqueues a Celery task.
- Celery worker + Redis: runs the LLM call, with retries and rate limits, then POSTs the result to the WordPress REST API.
- WordPress: updates the post or its metadata and exposes a status view.
The published prerequisites for that example are WordPress 6.x with Application Passwords enabled, Python 3.11+, Django 5.x, Celery 5.x, Redis, an OpenAI-compatible API key, and public HTTPS endpoints (ngrok for local testing, or a reverse proxy). Treat those versions as the documented environment of the reference implementation, not as a minimum you have independently verified.
Step 1: Enqueue the job from WordPress
The WordPress side is deliberately small: a menu page, a capability check, a nonce, and a function that sends the signed request. The published plugin skeleton lives at wp-content/plugins/ai-job-queue/ai-job-queue.php and follows this shape:
add_action('admin_menu', function () {
add_menu_page('AI Jobs', 'AI Jobs', 'edit_posts', 'ai-jobs', 'ai_jobs_page');
});
function ai_jobs_page() {
if (!current_user_can('edit_posts')) wp_die('No access');
if (isset($_POST['ai_enqueue']) && check_admin_referer('ai_jobs_nonce')) {
$post_id = intval($_POST['post_id']);
$resp = ai_send_job($post_id);
echo 'Queued: ' . esc_html($resp);
}
}Three details in that snippet carry most of the security weight. current_user_can('edit_posts') prevents lower-privileged users from spending your API budget. check_admin_referer('ai_jobs_nonce') blocks cross-site request forgery. intval() on the post ID keeps the payload numeric. The job body itself carries the post ID, the title, the plain-text content (the example strips tags before sending), and the callback URL the backend should post back to:
{
"post_id": 123,
"title": "Post title",
"content": "Plain-text body",
"callback_url": "https://example.com/wp-json/ai-jobs/v1/callback"
}Expected result: the backend accepts the request and returns an identifier for the job, which WordPress stores so the admin screen can display status later.
Step 2: Verify the signature and enqueue
The Django endpoint should do exactly two things before returning: verify that the request really came from your WordPress site, and hand the work to Celery. HMAC verification uses a shared secret that never travels in the request body. Returning immediately with a job ID is what keeps the WordPress admin responsive; the article is not ready yet, and the interface should say so rather than block.
Step 3: Let the worker call the model
The Celery worker is where retries and rate limits live. Concurrency should be set from the provider’s documented request limits, not from how many CPU cores the worker host has. A worker that fires twenty simultaneous requests at a provider with a five-request limit will spend its time handling 429 responses. Redis acts as the broker in this reference stack; other brokers exist, but the published example uses Redis.
Step 4: Return the result through a callback
When generation finishes, the worker POSTs the article back to the WordPress REST API, which updates the post or its metadata and updates the visible status. Authentication for that return trip is typically WordPress Application Passwords, which is why the reference prerequisites call for them to be enabled. If the callback fails, the job is not finished — it is finished-with-an-error, and it needs a retry path.
Idempotency: the property that keeps retries safe
Retries are the main reason queues are worth the operational cost, and also the main reason they can corrupt data. If a callback times out after WordPress has already written the post, a naive retry creates a duplicate. The fix is idempotency: every job carries a stable identifier, and the receiving endpoint checks whether that identifier has already been applied before writing anything. In practice this means storing the job ID alongside the post meta and treating a repeat callback as a no-op that returns success.
The same principle applies inside the worker. A task that is acknowledged before the model call completes will be lost if the worker is killed mid-request. Late acknowledgement, plus a bounded retry count and exponential backoff, keeps the queue honest about what actually happened.
Pacing, scheduling, and the calendar
A queue is a throttle, not just a buffer. The jekcms batch planner exposes this directly by spreading titles across days rather than publishing them at once. The reasoning is editorial as much as technical: a fifty-post dump is not a publishing strategy, and it also concentrates every failure and every review task into a single window. Spacing jobs gives you time to inspect the first outputs before the remaining forty-five are generated with the same prompt and model.
Where human review belongs
Nothing in the architecture above requires automatic publication, and the reference stack writes into WordPress rather than publishing. A queue that generates fifty drafts still produces fifty drafts that someone must check for factual accuracy, tone, and duplication. The practical pattern is to have the worker write into a draft or pending state, keep the job ID in post meta for traceability, and let an editor approve before anything goes live. The queue removes the mechanical work; it does not remove the editorial one.
Troubleshooting: symptoms and first checks
| Symptom | Likely cause | First check |
|---|---|---|
| Jobs stay queued and nothing runs | No worker is consuming the broker | Confirm the worker process is running and pointed at the same broker URL as the API |
| Callback returns 401 | Application Password revoked, or HMAC mismatch | Re-issue the Application Password; confirm both sides use the same secret and signing algorithm |
| Duplicate posts appear | Callback retried without an idempotency check | Look for a job ID stored in post meta and a guard that skips already-applied jobs |
| Provider returns 429 | Worker concurrency exceeds the provider’s rate limit | Lower concurrency and add backoff; check the provider’s documented limits |
| Jobs vanish after a worker restart | Task acknowledged before completion | Enable late acknowledgement and inspect the retry configuration |
Mask secrets in any log you share. HMAC keys, Application Passwords, and provider API keys should never appear in a troubleshooting thread.
Limitations and tradeoffs
A queue does not improve output quality; it only makes volume manageable. It also adds real operational surface: a broker to keep running, a worker host to monitor, a callback endpoint to secure, and a status view to maintain. If you need one article, the single-post path — AI Draft in jekcms, or a direct call from your workflow — is simpler and has fewer moving parts. Queues earn their complexity when the batch is large enough that partial failure, pacing, and per-job traceability actually matter.
There is also a cost dimension. Every retry consumes tokens again unless your worker caches results or checks state before regenerating. And if you run a local model instead of a hosted API, throughput is bounded by your own hardware, which usually means serializing jobs rather than parallelizing them.
A sensible first run
Start with five titles at low concurrency and a draft-only output state. Verify that the job count matches the title count, that each job ID appears in post meta, and that a deliberately failed callback does not create a duplicate when it is retried. Once that loop is boring, raise the batch size and the spread interval together, and keep the review checkpoint in place.
If you are wiring this up now, share your environment and the exact symptom you hit — broker, worker version, provider, and the error text — in the Ai-Forum.org community, so the next person building the same pipeline has a starting point.
FAQ
How many jobs can a single worker process at once?
That depends on your provider's rate limits and the latency of each generation call, not on the queue software itself. The published Celery example configures retries and rate limits for exactly this reason; set concurrency from the provider's documented limits and raise it only after observing error rates.
Do I have to use Redis and Celery?
No. Redis is the broker used in the published WordPress-to-Django reference architecture, and Celery is the worker framework in that example. Other brokers and worker systems exist, but the design principles — durable jobs, retries, idempotent callbacks — carry over regardless of the specific tools.
What happens if the callback to WordPress fails?
The job should be treated as failed rather than complete. A bounded retry with backoff is the usual fix, combined with an idempotency check on the receiving endpoint so that a callback which actually succeeded before timing out does not produce a duplicate post.
Should queued articles publish automatically?
The reference architecture writes results back into WordPress rather than publishing them, and there is a practical reason to keep it that way: generated drafts still need checking for factual accuracy, tone, and duplication. Writing to a draft or pending state and keeping the job ID in post meta preserves traceability while leaving the publish decision to an editor.