Evaluation tests for releases
I use a fixed set of evaluation tasks in continuous integration, with an explicit pass threshold. A release stops when results fall below it. The suite only covers its chosen tasks and cannot establish how every future run will behave.
How do evaluation tests check a release?
I use a fixed set of evaluation tasks in continuous integration, with an explicit pass threshold. A release stops when results fall below it.
Implementation
- Freeze a golden suite of tasks drawn from real workflows, with expected outputs reviewed by a human once.
- Run the suite in its own CI workflow, separate from unit tests, so a slow eval does not block ordinary commits.
- Add adversarial cases: prompt injection, contradictory instructions, missing tools, truncated context.
- Record per-task cost and latency alongside pass/fail so a regression in spend is visible.
- Set an explicit threshold and fail the workflow below it.
What can go wrong
- A suite containing only passing cases can miss failures outside its coverage.
- An unenforced threshold allows a release with failing evaluation results.
- Updating expected outputs without review can conceal a regression.
Checks to try
- Introduce a known-bad change and confirm the eval workflow fails.
- Check that the run cost is recorded per task and trending.
Questions this pattern answers
How do evaluation tests check a release?
I use a fixed set of evaluation tasks in continuous integration, with an explicit pass threshold. A release stops when results fall below it.
What steps does this method use?
Freeze a golden suite of tasks drawn from real workflows, with expected outputs reviewed by a human once. Run the suite in its own CI workflow, separate from unit tests, so a slow eval does not block ordinary commits. Add adversarial cases: prompt injection, contradictory instructions, missing tools, truncated context. Record per-task cost and latency alongside pass/fail so a regression in spend is visible. Set an explicit threshold and fail the workflow below it.
How can I check the result?
Introduce a known-bad change and confirm the eval workflow fails. Check that the run cost is recorded per task and trending.
What can go wrong with this approach?
A suite containing only passing cases can miss failures outside its coverage. An unenforced threshold allows a release with failing evaluation results. Updating expected outputs without review can conceal a regression.
Related projects
ClawKeeper →
ClawKeeper checks proposed Claude agent actions against explicit rules before a model call. Autonomous spend defaults to zero, and the database enforces an append-only audit log.
TechTide Harness Kit →
TechTide Harness Kit uses JSON Schema to declare skill permissions, risk tiers, network access, and approval requirements. Automated checks validate declarations and run prompt-injection fixtures.
CipherClaw →
CipherClaw analyzes recorded agent activity without model calls. It builds causal graphs, scores possible causes, and flags failure patterns.
Swarm 357 →
357 Claude agent roles across six business layers, with portable memory, enforced cost caps, a bash policy gate, and durable checkpoints.
Related patterns
Acceptance tests for agent work →
Executable checks test the task and detect regressions.
MCP tool permissions →
The server checks which tools a caller may use.
Recording agent runs →
A shared trace ID connects the records of a run.
Spending limits for agent runs →
Budget checks run before provider requests.
I explain this method here. Current service scope, prices, and terms are on TechTide AI.
Systems Decision Audit at TechTide AI→© 2026 Alex Cinovoj · TechTide AI · Columbus, OH