Acceptance tests for agent work
An agent run has no natural definition of done. Acceptance tests supply one: a fixed set of assertions that must hold after the run, written before the run, and executed by code rather than by reading the transcript.
What is acceptance tests for AI agents?
An agent run has no natural definition of done. Acceptance tests supply one: a fixed set of assertions that must hold after the run, written before the run, and executed by code rather than by reading the transcript.
Implementation
- Write the assertions as executable checks in the repository, next to the code the agent will touch.
- Separate task assertions (did the requested change happen) from invariant assertions (did anything else break).
- Keep every assertion deterministic. If a check needs a model to decide, it belongs in the eval suite, not in acceptance.
- Fail the run on any invariant break, even when the task assertion passes.
- Store the pass/fail record with the trace ID so the result is attributable to a specific run.
{
"run_id": "run_2f9c...",
"task": "add rate limit to /api/chat",
"task_assertions": { "passed": 4, "failed": 0 },
"invariant_assertions": { "passed": 128, "failed": 0 },
"verdict": "accept",
"recorded_at": "2026-08-06T18:04:11Z"
}Failure modes
- Assertions written after the run, which encode whatever the agent produced.
- Model-graded acceptance, which drifts between runs and cannot be audited.
- Task assertions with no invariant coverage, so regressions ship green.
How it is verified
- Re-run the suite on the pre-change commit and confirm it fails for the right reason.
- Confirm each record links to a retrievable trace.
Questions this pattern answers
What is acceptance tests for AI agents?
An agent run has no natural definition of done. Acceptance tests supply one: a fixed set of assertions that must hold after the run, written before the run, and executed by code rather than by reading the transcript.
How do you implement acceptance tests for agent work?
Write the assertions as executable checks in the repository, next to the code the agent will touch. Separate task assertions (did the requested change happen) from invariant assertions (did anything else break). Keep every assertion deterministic. If a check needs a model to decide, it belongs in the eval suite, not in acceptance. Fail the run on any invariant break, even when the task assertion passes. Store the pass/fail record with the trace ID so the result is attributable to a specific run.
How is acceptance tests for agent work verified?
Re-run the suite on the pre-change commit and confirm it fails for the right reason. Confirm each record links to a retrievable trace.
What usually goes wrong with acceptance tests for agent work?
Assertions written after the run, which encode whatever the agent produced. Model-graded acceptance, which drifts between runs and cannot be audited. Task assertions with no invariant coverage, so regressions ship green.
Projects in this track
ClawKeeper →
Governed Claude agent control plane. A deterministic policy engine runs before every LLM call, autonomous spend defaults to zero, and the audit log is enforced by the database.
TechTide Harness Kit →
Trust contracts for agent skills expressed as JSON Schema: risk tiers, tool permissions, deny-by-default egress, approval gates, and prompt-injection fixtures behind a wall of CI validation gates.
CipherClaw →
Deterministic multi-agent debugger with zero LLM calls in the analysis path: causal graphs, weighted root-cause scoring, and failure prediction.
Swarm 357 →
357 Claude agent roles across six business layers, with portable memory, enforced cost caps, a bash policy gate, and durable checkpoints.
Related patterns
This page documents how the pattern is built. Engagement scope and pricing live on TechTide AI.
AI Production Systems at TechTide AI→© 2026 Alex Cinovoj · TechTide AI · Columbus, OH