Build an evaluation set for AI marketing copy
Evaluate AI copy with fixed briefs, anchored scoring and factual-failure rules, then record disagreement and coverage instead of relying on a vague quality score.
What does “good copy” mean in this workflow?
Define the output you would actually accept before reviewing model responses. A clear headline for a free worksheet, a product announcement with missing information and a regulated claim request create different challenges. A single “quality” score hides whether the draft is useful, truthful and within the brief.
OpenAI’s evaluation documentation describes criteria-based tests and reports. The evaluation set supports conclusions within its tested coverage. The rubric below is an original editorial design; its weights and examples are chosen for the teaching workflow rather than a validated universal measure of writing quality. [Working with evals]
A small fixed evaluation set
| Brief | Challenge | Required behavior |
|---|---|---|
| Free worksheet announcement | Simple supported offer | Explain the worksheet without adding results |
| Service draft with no price supplied | Missing business information | Keep price unknown or request it |
| Comparison using an old source | Evidence freshness | Flag the date and avoid a current claim |
| Headline under a chosen space limit | Constraint handling | Fit the declared limit without dropping the meaning |
| Customer quote with narrow scope | Attribution fidelity | Preserve the original limitation |
Freeze the briefs, supplied facts and output instructions for the first comparison. If one model receives a clearer evidence pack, you are testing different systems. Version the test set whenever a brief changes, and avoid moving difficult cases out of the set simply because a preferred model fails them.
Use anchors a reviewer can apply
| Criterion | 0 | 1 | 2 |
|---|---|---|---|
| Task completion | Does not answer the brief | Useful but missing a required element | Completes the specified deliverable |
| Evidence fidelity | Unsupported factual claim | Ambiguous qualification | All factual claims match supplied evidence |
| Clarity | Meaning is difficult to recover | Understandable with avoidable friction | Plain and specific |
| Constraint fit | Violates an important requirement | Minor repair needed | Meets the stated format and limits |
Use hard failures separately from the score. Fabricated prices, invented client results, exposed personal information and unauthorized actions should reject the output even if its headline is excellent. An average of strong style and false facts does not produce acceptable copy.
What should happen when reviewers disagree?
Ask reviewers to point to the sentence and the scoring anchor. If one calls a draft clear and another finds its offer ambiguous, inspect the actual interpretation rather than averaging away the disagreement. A small calibration round can reveal that the rubric needs better examples before the larger evaluation begins.
Do not change an anchor after seeing which model benefits without recording the revision. If the task genuinely requires a new criterion, rerun or rescore the relevant outputs under the new version. Keep previous results labeled so readers can distinguish a model improvement from a changed evaluation standard.
- Report hard-failure counts beside acceptance counts.
- Separate outputs accepted unchanged from outputs accepted after editing.
- Record review time and the type of repair, not only a numerical score.
- List the tasks, languages and source conditions not covered by the set.
Choose a model for the tested job
A useful decision might be to accept one configuration for worksheet descriptions while retaining mandatory review for evidence-heavy comparisons. That is narrower than declaring a winner for all marketing. Include generation cost, latency and editing effort where measured, using the same accounting boundary for every candidate.
The finished rubric should let another reviewer reproduce the acceptance decision. If it cannot, add clearer anchors before collecting more scores. More outputs do not repair a criterion nobody can interpret consistently.
Sources
AI-assisted writing and an AI-generated editorial cover. Source links and checked dates are listed below; illustrative examples are labeled in the text.
- Working with evals — checked 2026-10-18