By Scott Sumner | AI at Work | Published October 6, 2026

A businesswoman compares sample sheets beside a purple stamp, illustrating the need to retest existing jobs when an AI workflow changes.

An AI upgrade can improve the demo and still make an existing task worse. Before changing a model, prompt or connected knowledge source, keep a small set of examples that show what already works.

AI regression testing checks whether an updated AI workflow still handles tasks it previously performed correctly. Run saved examples against the old and new versions, judge them against explicit requirements, and investigate failures before releasing the change.

You do not need a research department to start. You need a few representative jobs, clear expectations and enough discipline to keep the awkward examples instead of deleting them.

Jump ahead

Why a better model can be worse at your task

Imagine an agency using AI to turn approved project notes into a first-draft scope of work. Someone changes the prompt to make the writing shorter. The next draft reads beautifully, but it drops the sentence that says content entry is not included.

The edit improved the prose. It weakened the scope.

That is a hypothetical example, not a client incident. It explains why checking a new feature is different from checking that existing behavior survived.

Anthropic’s evaluation guidance distinguishes capability evaluations, which ask what a system can do, from regression evaluations, which ask whether it still handles tasks it used to handle. That distinction is useful far beyond engineering teams.

A public benchmark can help you shortlist models. It cannot tell you whether your agency’s exclusions survive a rewrite. Your own test cases have to answer that question.

The thing being tested is also bigger than the model. Instructions, source documents, retrieval, tools and application code can all affect the result. Record what changed instead of automatically blaming the chatbot.

Build a small test folder from real work

I would start with 12 cases for one narrowly defined workflow. That is a practical starting point, not a statistically validated sample size or proof of safety. A high-risk or widely used system needs substantially more coverage.

For a scope-drafting assistant, a starter collection could include:

  • Five ordinary briefs with the information needed to draft a scope.
  • Three incomplete briefs that should trigger questions rather than invented details.
  • Two briefs with exceptions, such as a service that is usually included but is explicitly excluded here.
  • Two previously troublesome examples that the team has already corrected.

Use approved, de-identified examples where possible. Keep customer secrets out of a casual testing spreadsheet. Your workspace and data-retention choices still apply during testing.

Each case needs the input, the source material available to the assistant, and the expected behavior. Save the source version too. A price list that changes between runs makes a comparison harder to interpret.

Do not collect only the easy jobs. A folder full of neat briefs tests whether the assistant can handle neat briefs. Customers rarely organize their questions for your convenience.

Decide what passes before seeing the answer

“Looks good” is not a useful release rule. Write down what must remain true.

For the hypothetical scope workflow, the rubric might require that the draft retains every stated exclusion, uses only approved prices, preserves the requested delivery dates and flags missing information. Tone can be assessed separately.

One polished paragraph should not cancel out an invented price. Mark material errors as failures rather than hiding them inside an average writing score.

OpenAI’s evaluation guidance recommends defining objectives, collecting task-relevant data, defining metrics, comparing results and evaluating continuously. It also recommends combining metrics with human judgment.

The check should fit the requirement:

  • Exact facts: compare prices, identifiers and dates against the approved source.
  • Required meaning: have a knowledgeable reviewer confirm that exclusions and commitments survived.
  • Behavior under uncertainty: check whether the assistant asks for a missing fact instead of making one up.
  • Tool outcomes: if the workflow changes a record, inspect the resulting record in a test environment. The assistant saying “done” is not the same as the change being correct.

Anthropic makes that last distinction explicitly: the final outcome is separate from what an agent says in its conversation. Test systems that send messages or modify data in a sandbox, with real recipients and production writes disabled.

A second AI can help sort or flag answers. It should not become an unquestioned examiner. Both sources discuss calibrating model-based grading against human judgments.

Compare changes without moving every variable

Save the current prompt and configuration before editing them. Run the same cases against the current and proposed versions, with the same source material and comparable settings.

If you change the model, rewrite the prompt and replace the knowledge base in one afternoon, a better result will be hard to explain. A worse result will be harder to fix.

Keep a simple record for each run: case ID, version, output, pass or fail, failed requirement and reviewer note. Add run time or cost if those matter to the job, but do not let a cheaper answer excuse an unacceptable error. The broader cost-per-finished-job view remains useful alongside quality checks.

Repeat important and unstable cases. OpenAI notes that generative AI can produce different outputs for the same input, and Anthropic describes running multiple trials to account for that variability. One successful response is weak evidence of dependable behavior.

Keep some examples out of the prompt-tuning process. If every edit is designed around the same visible cases, you can end up with a system that passes its familiar test but fails on ordinary new work. Add fresh cases from actual use, with appropriate privacy controls.

Make a release decision, not just a score

For this scope-drafting example, my release rule would be simple: do not accept a change that introduces a material pricing, scope or deadline error in the test set. That is my recommended business rule, not a universal threshold supplied by a model vendor.

If the revised version fails, inspect the failure before making the prompt longer. Was the source missing? Did retrieval return the wrong document? Did the model omit a requirement? Is the test itself demanding something the brief does not support?

Keep a rollback path. That may mean restoring a saved prompt, choosing the prior model version where available, or temporarily returning the task to manual review. Do not assume a provider will keep every old model available indefinitely.

Passing the folder is not the end. Continue reviewing live results, collect new failures and turn them into future tests. Retest after meaningful changes to the prompt, model, documents, tools or business rules.

The payoff is a more concrete conversation. Instead of “the AI feels worse,” you can say, “the new version dropped this exclusion on this saved brief.” Someone can act on that.

What to do next

Save five corrected examples from one recurring AI task this week, together with the requirement each correction protects. That is the beginning of a useful test folder.

If AI is becoming part of your website or customer-facing process, talk with Scott Sumner about reviewing the workflow. The useful question is whether the system does the job reliably, not whether the latest demo looks impressive.

Sources

This is an evergreen operational guide, not a report of a new product launch. Sources checked October 6, 2026. The scope-drafting scenario, 12-case starting set and suggested release rule are illustrative recommendations.