Replaying and comparing agent traces
I compare recorded runs step by step to find where they first differ. Replay uses the stored inputs for deterministic steps, without calling a provider again. Trace completeness and import validation limit what can be reproduced.
How do I compare recorded agent runs?
I compare recorded runs step by step to find where they first differ. Replay uses the stored inputs for deterministic steps, without calling a provider again.
Implementation
- Store enough per-step input to re-execute the deterministic parts of a run without a provider.
- Treat imported traces as untrusted input and validate them against a schema before storage.
- Diff two runs step by step and report the first divergent step, not a similarity score.
- Keep redaction applied through the replay path.
What can go wrong
- Calling a provider during replay can produce different output from the recorded run.
- Comparing only final output can hide where the runs first differed.
Checks to try
- Replay a stored run and confirm the deterministic steps reproduce exactly.
- Feed a malformed trace to the importer and confirm it is rejected.
Questions this pattern answers
How do I compare recorded agent runs?
I compare recorded runs step by step to find where they first differ. Replay uses the stored inputs for deterministic steps, without calling a provider again.
What steps does this method use?
Store enough per-step input to re-execute the deterministic parts of a run without a provider. Treat imported traces as untrusted input and validate them against a schema before storage. Diff two runs step by step and report the first divergent step, not a similarity score. Keep redaction applied through the replay path.
How can I check the result?
Replay a stored run and confirm the deterministic steps reproduce exactly. Feed a malformed trace to the importer and confirm it is rejected.
What can go wrong with this approach?
Calling a provider during replay can produce different output from the recorded run. Comparing only final output can hide where the runs first differed.
Related projects
ClawKeeper →
ClawKeeper checks proposed Claude agent actions against explicit rules before a model call. Autonomous spend defaults to zero, and the database enforces an append-only audit log.
TechTide Harness Kit →
TechTide Harness Kit uses JSON Schema to declare skill permissions, risk tiers, network access, and approval requirements. Automated checks validate declarations and run prompt-injection fixtures.
CipherClaw →
CipherClaw analyzes recorded agent activity without model calls. It builds causal graphs, scores possible causes, and flags failure patterns.
Swarm 357 →
357 Claude agent roles across six business layers, with portable memory, enforced cost caps, a bash policy gate, and durable checkpoints.
Related patterns
Acceptance tests for agent work →
Executable checks test the task and detect regressions.
Evaluation tests for releases →
A fixed evaluation suite checks release requirements.
MCP tool permissions →
The server checks which tools a caller may use.
Recording agent runs →
A shared trace ID connects the records of a run.
I explain this method here. Current service scope, prices, and terms are on TechTide AI.
Systems Decision Audit at TechTide AI→© 2026 Alex Cinovoj · TechTide AI · Columbus, OH