Decide what good means before release.

The business helps define the test

OpenAI’s Morgan Stanley story describes experts grading outputs and repeated evaluations across specific tasks. Its meeting follow-up workflow also keeps review with the advisor before information is finalized or sent.

Deploy Co. makes business ownership of evaluation criteria central to go-live confidence. Our adaptation is a small acceptance scorecard agreed with the person accountable for the workflow, using ordinary examples and meaningful exceptions.

Check what actually changed

Anthropic’s agent-evaluation guidance distinguishes a completed task from a convincing transcript. The test should inspect the resulting artifact or system state, and account for variation across repeated runs.

For document review, that might mean correct fields, supporting source references and an honest exception for missing evidence. For intake, it might mean a complete record and a reply that follows the business’s policy. A tool call returning successfully is only one part of that result.

• Use representative work, incomplete inputs and plausible failures.
• Check permissions, tool errors and unintended or duplicated actions.
• Keep regression cases for changes to models, prompts, sources and integrations.
• Record unresolved misses and the person who owns the release decision.

Make the economics honest

The Zendesk customer story emphasizes evaluating realistic customer tasks and cost per completed task. Deploy Co.’s monitoring guidance also pairs efficiency with the quality it might compromise.

Our proposed scorecard includes time spent checking, correcting and recovering from failures. Compare that total with the current process, then include usage and maintenance costs. A faster draft is useful only if the whole workflow is better for the team.