Content Evals: A Test Suite for AI Output

Ask AI →

A content team ships a prompt change on a Tuesday. Nothing breaks. No 500 errors, no missing pages, no alert in Slack. Three weeks later newsletter replies are down, three posts have been rewritten by sales, and nobody can name the edit that caused it.

That’s how AI content fails. It doesn’t crash. It drifts.

Normal QA catches broken things. Content evals catch degraded things. The idea comes from software: if you can’t test a system, you can’t change it safely. A language model with a long prompt, a brand guide, and five data sources is a system with a hundred inputs. The only way to change it on purpose is to hold a fixed exam and grade every change against it.

This is the build guide: four parts, seven codes, one prompt, and the gate.

Why AI content needs a test suite

A style guide describes what good looks like. It doesn’t test whether the draft in front of you is good. That gap was survivable when a human wrote every word, because the writer carried the standard in their head. It stops being survivable when the words come from a model that regenerates them slightly differently every time.

Every change to your content system moves the output distribution. A new prompt paragraph. A model upgrade. A new writer. A new source document. Some changes are improvements. Some are regressions. Without a fixed test set, you can’t tell which one you just shipped. Neither can your editor, because humans are inconsistent graders under deadline.

Hamel Husain makes the case in Your AI Product Needs Evals that evals are unit tests for a probabilistic system. You don’t test one output. You test a sample against a standard you defined in advance. That is the whole discipline, and it transfers to content almost unchanged.

A judge that agrees with your taste four times out of five catches most drift without a human reading every draft. The fifth time is why a person still signs off on anything that blocks publication.

The four parts of a content eval system

Teams overbuild this. You don’t need a data platform, a vector database, or a new hire. You need four artifacts, and each one has a job the others cannot do.

If you have read the content engineering feedback loop, this is the evaluation stage built out. That piece places evaluation between generation and distribution. This is what that stage contains.

Build the golden set

The golden set is a frozen sample of real work with human grades attached. Not aspirational examples. Real published pieces, including the ones that underwhelmed. Four steps, one afternoon.

1. 1
   Pull 50 pieces across content types

   Twelve to fifteen each from how-to, framework, trend, and case study. A set that is 80% how-to will pass a prompt that breaks your opinion writing.
2. 2
   Add ten known-bad examples

   Killed drafts. Pages that lost rankings after an update. Copy that sales quietly rewrote before sending. A test set with no failures can’t detect a regression. It can only confirm that everything still looks fine.
3. 3
   Grade each one by hand

   Pass, pass with edits, or fail. Then write the reason using the failure codes from the next section. The reason is the valuable part. It becomes the standard the judge is measured against.
4. 4
   Freeze it and version it

   Store the pieces, the grades, and the reasons in one file. Add cases when you add a content type. Never edit an old case to make a new prompt look better. That is how teams end up with a test suite that always passes and a blog nobody reads.

Write the rubric as failure codes, not adjectives

“The draft is not good” cannot be tested. “FACT: the claim has no source in the brief” can. That translation is the entire trick, and it is why most content rubrics fail. They list virtues. Virtues cannot be scored by a model or a freelancer at 5pm on a Friday.

Seven codes cover most of what goes wrong in B2B content. Add an eighth only when you find a defect that none of these seven can describe.

Grade correctness first and style last, because a beautiful piece with a wrong number is a fail. The order also protects you from the most common judge failure, where a model rewards fluent prose and misses a fabricated statistic. Put FACT and SOURCE ahead of BRAND and TONE in the prompt.

Calibrate the judge against your own taste

A judge is a model with a prompt, not an oracle. Two design choices decide whether it is useful. First, use pairwise comparison instead of a 1 to 10 score, because models cluster at 7 and the scale tells you nothing. Show a known pass and a known fail, then ask which side of the line the draft sits on. Second, anchor every code to a test question, so the model has something falsifiable to check.

Then calibrate. Hand-label twenty pairs yourself. Run the judge on the same twenty. Compare. Below 80% agreement, the rubric is ambiguous or the prompt is underspecified, and no amount of model swapping will fix it. Eugene Yan’s writing on LLM evaluators covers the patterns worth stealing, including position bias and the value of reference examples.

The 80% target traces to Zheng and colleagues, who measured strong models against human preference. Agreement cleared 80%, with known biases toward verbosity, position, and self-enhancement. Design around those biases instead of pretending they do not exist.

You are a content evaluator for a B2B publication.

INPUTS
- BRIEF: the claims, sources, audience, and CTA the writer was given
- DRAFT: the piece to grade
- PASS_EXAMPLE and FAIL_EXAMPLE: labeled references for this content type

TASK
Grade the DRAFT against the seven codes below. For each code, return
pass or fail and one verbatim quote from the DRAFT as evidence. If you
cannot find a quote, return fail with evidence "no supporting text".

CODES
1. FACT: every number, date, and named claim traceable to the BRIEF
2. SOURCE: citation exists and supports the sentence it sits under
3. BRAND: voice rules followed, banned words absent
4. STRUCT: answer present in the first 80 words and in the TL;DR
5. CLAIM: no superlative, guarantee, or absolute claim
6. CTA: correct offer, correct link, exactly one ask
7. TONE: no condescension, hype, or hedging

RULES
- Check FACT and SOURCE before style. Fluent prose is not evidence.
- Do not reward length, confidence, or specificity that the BRIEF lacks.
- Do not infer facts. If it is not in the BRIEF, the DRAFT cannot assert it.
- Do not rewrite the DRAFT.

OUTPUT
JSON only:
{"codes":[{"code":"FACT","result":"pass|fail","evidence":"quote"}],
 "overall":"block|warn|ship"}

Recalibrate on three triggers: a model change, a rubric change, and any month where agreement drifts more than five points. For regulated content, run two judges and escalate disagreement to a human. Two models disagreeing is a useful signal. One model being confident is not.

Put the gate where the work happens

A test suite nobody runs is documentation. The gate has to sit between the draft and the publish button, with teeth in proportion to the damage a defect causes.

Wire the gate in three places. First, in the CMS or the publishing script, so no draft bypasses it. Second, on every batch run, so a bad day in the pipeline is visible before a week of content is queued. Third, on every prompt or model change, running the full golden set before the change is merged. That third one turns evals into infrastructure rather than a report.

Fifty pieces at a couple thousand tokens each is a coffee budget, and it runs in minutes. Compare that with the cost of a wrong statistic on a page that sales sends to fifty prospects. The same logic that made scaling quality without headcount possible applies here. The system carries the standard, and people spend their attention on the exceptions.

Track five numbers, ignore the rest

A dashboard with forty metrics gets opened once. Five numbers change behavior.

The golden-set pass rate tells you whether a change helped. It should hold steady or rise after every prompt edit. Escaped defects per ten pieces tell you whether the gate catches the right things.

The other three describe the system. Judge-human agreement tells you whether the judge is still trustworthy. Revision time per piece tells you whether the system is saving the editor or feeding them more work. Failure-code distribution tells you what to fix upstream, in the brief rather than the draft.

That last number is the one teams underuse. If FACT fails on 30% of drafts, the problem is neither the writer nor the model. The brief doesn’t contain the sources. Fix the brief template once and 30% of your defects disappear.

The system should improve the input, not just grade the output. That is the difference between a proofreader with a checklist and a content engineer with a quality system. What is still rare is the discipline to test the machine before trusting it with the byline.

Start with ten pieces, four codes, and one judge prompt. Run it on the next batch you produce. You will find a defect in the first twenty minutes, and that defect was going to publish.