Acceptance tests for agent work
I write acceptance checks before an agent starts. Task assertions check the requested change, while invariant assertions check what must remain unchanged. The checks run in code, and their result is recorded with the run’s trace ID.
How do I check whether an agent completed its task?
I write acceptance checks before an agent starts. Task assertions check the requested change, while invariant assertions check what must remain unchanged.
Implementation
- Write the assertions as executable checks in the repository, next to the code the agent will touch.
- Separate task assertions (did the requested change happen) from invariant assertions (did anything else break).
- Keep every assertion deterministic. If a check needs a model to decide, it belongs in the eval suite, not in acceptance.
- Fail the run on any invariant break, even when the task assertion passes.
- Store the pass/fail record with the trace ID so the result is attributable to a specific run.
{
"run_id": "run_2f9c...",
"task": "add rate limit to /api/chat",
"task_assertions": { "passed": 4, "failed": 0 },
"invariant_assertions": { "passed": 128, "failed": 0 },
"verdict": "accept",
"recorded_at": "2026-08-06T18:04:11Z"
}What can go wrong
- Assertions written after a run can merely reproduce the agent’s output.
- Model-graded acceptance can vary across runs and is harder to audit.
- Task assertions without regression checks can pass while unrelated behavior breaks.
Checks to try
- Re-run the suite on the pre-change commit and confirm it fails for the right reason.
- Confirm each record links to a retrievable trace.
Questions this pattern answers
How do I check whether an agent completed its task?
I write acceptance checks before an agent starts. Task assertions check the requested change, while invariant assertions check what must remain unchanged.
What steps does this method use?
Write the assertions as executable checks in the repository, next to the code the agent will touch. Separate task assertions (did the requested change happen) from invariant assertions (did anything else break). Keep every assertion deterministic. If a check needs a model to decide, it belongs in the eval suite, not in acceptance. Fail the run on any invariant break, even when the task assertion passes. Store the pass/fail record with the trace ID so the result is attributable to a specific run.
How can I check the result?
Re-run the suite on the pre-change commit and confirm it fails for the right reason. Confirm each record links to a retrievable trace.
What can go wrong with this approach?
Assertions written after a run can merely reproduce the agent’s output. Model-graded acceptance can vary across runs and is harder to audit. Task assertions without regression checks can pass while unrelated behavior breaks.
Related projects
ClawKeeper →
ClawKeeper checks proposed Claude agent actions against explicit rules before a model call. Autonomous spend defaults to zero, and the database enforces an append-only audit log.
TechTide Harness Kit →
TechTide Harness Kit uses JSON Schema to declare skill permissions, risk tiers, network access, and approval requirements. Automated checks validate declarations and run prompt-injection fixtures.
CipherClaw →
CipherClaw analyzes recorded agent activity without model calls. It builds causal graphs, scores possible causes, and flags failure patterns.
Swarm 357 →
357 Claude agent roles across six business layers, with portable memory, enforced cost caps, a bash policy gate, and durable checkpoints.
Related patterns
I explain this method here. Current service scope, prices, and terms are on TechTide AI.
Systems Decision Audit at TechTide AI→© 2026 Alex Cinovoj · TechTide AI · Columbus, OH