AI adoption
Keep a small test set for your AI workflow.
Keep representative examples and clear acceptance checks so you can compare changes to an AI workflow without relying on a memorable demonstration.
A hypothetical operations team improves the instructions used to turn job notes into customer updates. The next draft reads more naturally, so the new version is shared. Nobody checks whether it still handles an incomplete record or a cancelled appointment correctly. One attractive answer has become the evidence for changing the whole workflow. A small, maintained test set gives the team a more useful basis for that decision.
Choose examples that expose different problems
Start with a manageable set of synthetic cases that reflect the work. Include an ordinary complete record, a missing detail, conflicting information and an exception that should stop the normal process. Choose cases because they reveal something different, rather than filling a folder with near-identical examples.
For the customer-update workflow, one case might have a confirmed appointment, another an unconfirmed date and another a cancellation. Keep the inputs fixed when comparing versions. If both the instructions and the source information change, it becomes harder to understand what caused a different result.
Write down what a usable result must do
Describe the expected behaviour before running the test. Exact wording rarely needs to match, but the important facts, boundaries and next step should be clear. Separate essential requirements from preferences about tone or style.
- Use only the appointment details confirmed in the source.
- Make missing or conflicting information visible.
- Avoid commitments the business has not authorised.
- Follow the agreed route when the case cannot be completed.
- Produce an output the next person can use without reconstructing it.
Compare the current version with the proposed change
Run both versions against the same cases and judge them against the same checks. Record the instructions, tool or model label available to you, date, outputs and reviewer decisions. Keep these test records free of real customer information by using synthetic inputs throughout.
Repeat consequential cases to see whether a good result is consistent. A small set cannot establish reliability across every future task, and a repeated success does not remove the need for the workflow’s normal review. Record failures alongside successes. One invented appointment date matters more than several improvements in phrasing.
Let the evidence guide the next change
Decide whether to adopt the proposed version, revise it or retain the current one. If a critical case fails, hold the change for investigation. Do not quietly weaken the acceptance check to make the new version pass; a genuine change in requirements needs its own reason and decision.
Add a new synthetic case when real use reveals a gap, without copying the customer’s record into the test collection. Keep an owner and a current version so colleagues know which examples to use. Rerun the relevant cases when instructions, source structure, business rules or the AI tool change.
Rethink’s AI working sessions can help define these examples and checks around one real business task. Ongoing adoption support can help teams maintain them as the work evolves.