How to Create Regression Tests for AI-Generated WordPress Content

Regression testing AI-generated WordPress content works differently from testing ordinary PHP. An exact-match assertion such as assertSame() cannot grade a model’s output, because the correct answer this week may be worded differently from the correct answer last week. The practical replacement is an eval: a scored check that runs a real prompt, compares the output against a rubric, and reports how many cases still meet the bar after a prompt change, a provider switch, or simply the passage of time.

This guide is for WordPress developers and plugin authors who already have an AI-calling ability returning real output and now need to know when that output starts drifting. It covers the split between deterministic validation and scored evaluation, how to make output tractable to score with structured JSON, how to write field-level assertions instead of vague ones, and how to organize the whole thing so it does not slow down your PHPUnit suite.

Why exact-match assertions break on model output

A PHPUnit test asserts that one value is exactly equal to another. That works for the deterministic half of an AI feature: validating input, handling errors, and shaping the result into the structure your plugin expects. It stops working the moment you point the test at the model itself.

Writing evals for WordPress AI output regressions makes the distinction directly: assertSame() cannot grade model output, because the correct output this week might be worded differently from the correct output last week. An eval can, because it scores output against criteria instead of comparing it to one fixed string. The mental model shifts from “did this test pass” to “how many of these N cases still meet the bar, and did that number just drop.”

This is not a stricter test. It is a different kind of check, and treating it like a stricter PHPUnit test, expecting 100% pass every time, will produce a suite that fails constantly for reasons that are not regressions.

The two layers: deterministic validation and scored evaluation

Most AI-calling abilities in WordPress have two distinct halves, and they need two distinct testing strategies.

The first half is deterministic and belongs in PHPUnit: argument validation, error handling, capability checks, and the shaping of the model’s response into whatever your code consumes. This half can and should be tested with fakes and exact assertions.

The second half is the one a fake summarizer deliberately does not cover: whether the real model, given a real prompt, still produces good output after you have changed the prompt, switched providers, or just watched a few weeks pass. Fakes cannot answer that question, because a fake returns whatever you told it to return. Only a run against the real model can.

If you have already separated these concerns through dependency injection, you are in a good position: the same boundary you used to inject a fake is the boundary you use to inject the real client for an eval run.

Prerequisites before writing your first eval

The documented prerequisites for building an eval around a WordPress AI ability are narrow and worth checking off first:

• A dependency-injection pattern that lets the same ability run with either a fake or the real client.
• Structured output available through as_json_response(), so the model returns a shape you can inspect field by field.
• A working wp_ai_client_prompt() call that already returns real output.

Without structured output, scoring becomes guesswork. Without a working real call, there is nothing to score.

Make the output scoreable with structured JSON

The single biggest practical lever on eval quality is the shape of the model’s response. Structured output is easy to check field by field; free-form prose is not. A paragraph of generated copy can only be judged by another model or by a human, both of which are expensive and inconsistent at scale. A JSON object with a title, a categories array, a tags array, and a body string can be checked by ordinary PHP.

This is why as_json_response() matters for evals, not just for prompt engineering. It converts an unbounded grading problem into a set of bounded, testable assertions.

Write eval cases with real assertions per field

An eval case should assert something concrete about each field it cares about. Examples of assertions that are actually checkable:

• Category membership: the returned category is one of the categories that exist in the site.
• Tag counts: between two and five tags were returned, and none are duplicates.
• Length bounds: the body falls within a minimum and maximum word count.
• Required fields: every field your downstream code reads is present and non-empty.

What to avoid is the assertion that cannot fail usefully: “does this look right.” A criterion that a human cannot apply consistently is a criterion a script cannot apply at all.

Note that some criteria are legitimately variable between runs. A category assignment might be correct as either of two plausible values. Design the assertion to accept the set of acceptable outcomes rather than one exact value.

Organize suites, cases, evaluators, and tags

The wp-ai-evals package documents a small vocabulary that maps cleanly onto this problem. A suite is a namespaced collection owned by one plugin or feature. A case combines one input, a task, an expected value, an evaluator set, and a metadata record. A task invokes the system under test. An evaluator scores task output from 0 to 1 and returns pass/fail plus a reason. A tag is an orthogonal selector such as smoke, safety, quality, or slow.

The hierarchy is suite → case. Tags provide flexible subsets without forcing each case into a single group, which means you can run only the smoke cases after a small prompt edit and the full quality set before a release.

Small plugins can register suites directly during wp_ai_evals_init. Larger plugins can load a directory:

add_action( 'wp_ai_evals_init', static fn(Registry $registry) => $registry->load_directory(__DIR__ . '/suites') );

The loader accepts standalone PHP files that return a Suite, or modular directories where a suite.php manifest sits alongside a cases/ folder. Case files may return one EvaluationCase or an iterable of cases, and they load in stable lexical order with normal PHP access to your plugin’s callbacks, tasks, and custom evaluators.

Datasets expand into stable cases with ordinary iterables and Suite::add_cases(). A case built from a data row looks roughly like this:

$suite->add_cases(
    (static function () use ($rows): iterable {
        foreach ($rows as $id => $row) {
            yield EvaluationCase::make($id)
                ->input($row['input'])
                ->task(static fn(string $input) => my_plugin_answer($input))
                ->expected($row['expected'])
                ->evaluate_with(new ContainsText())
                ->tag('dataset', 'quality');
        }
    })()
);

Two design choices in that example are worth copying. First, the case ID comes from a stable data key rather than an array index, so results stay comparable across runs. Second, the evaluator is a named class rather than an inline closure, so the same scoring logic can be reused and reasoned about.

Keep evals out of the commit path

Evals cost real money and take real time. They call a live model, which means network latency, token costs, and occasional provider-side failures that have nothing to do with your code. The guidance is explicit: run them separately, not on every commit.

A workable split for a WordPress plugin looks like this:

1. On every commit: the PHPUnit suite, with fakes. Fast, free, deterministic.
2. On demand or on a schedule: the eval suite against the real model. Slower, costs tokens, results are scores rather than pass/fail.
3. Before a release or after a prompt change: the full eval suite, compared against the previous run.

The comparison step is the part that turns evals into regression tests. A single run tells you the current score. Two runs tell you whether something changed.

An illustrative scenario

Consider a plugin that generates post excerpts and assigns categories. Its eval suite holds twenty cases, each with a source post and a set of acceptable categories. A developer rewrites the system prompt to shorten excerpts. The PHPUnit suite still passes, because the fake client is unaffected. The eval run, however, shows the average score dropping from nineteen of twenty cases meeting the bar to fifteen, with the failures concentrated in cases where the shortened excerpt no longer mentions the assigned category. That drop is the regression signal, and it points directly at the interaction between the length constraint and the category criterion.

This scenario is illustrative rather than a reported test result. The numbers are chosen to show the shape of the signal, not to describe a measured run.

Reading results as a trend, not a verdict

Because evaluators return a score from 0 to 1 alongside pass/fail and a reason, the useful output of an eval run is a distribution and a delta. Store the results of each run so you can answer: which cases moved, in which direction, and what changed in the prompt or provider at the same time.

Two failure modes are common. The first is treating any score below 1.0 as a bug, which produces alert fatigue and eventually causes the team to ignore the suite. The second is never looking at the raw reasons, which hides the fact that a case is failing for a reason the rubric did not anticipate.

Limitations and tradeoffs

Evals are not a substitute for human review of published content. They check the properties you thought to encode; they do not check the properties you forgot. A suite that verifies category membership, tag counts, and length bounds says nothing about factual accuracy, tone, or whether the excerpt misrepresents the post.

Structured output also constrains the model. Forcing a JSON shape can reduce the fluency of generated prose, so the same structured response that makes scoring tractable may make the output slightly worse for readers. That tradeoff is worth measuring rather than assuming.

Finally, eval results are only as stable as the model behind them. Switching providers, changing model versions, or altering sampling settings can move scores without any change to your code. Record the model identifier and configuration alongside each run, or you will not be able to tell a code regression from a model change.

Next steps

Start with one ability, not all of them. Add as_json_response() if it is not already there, write five cases with concrete per-field assertions, tag them smoke, and run them manually against the real model. Record the scores. Then change the prompt deliberately and run again. If the second run shows a drop you can explain, your eval suite is working.

If you are setting this up in your own plugin and hit a wall, the Ai-Forum.org community is a reasonable place to describe your environment, the ability you are testing, the assertions you wrote, and the result you got. Include your WordPress version, the model or provider, and the exact evaluator output rather than a paraphrase, so others can reproduce the run.