Recovering runs from checkpoints
I store completed steps and their results so a run can resume after an interruption. Recovery also needs repeatable steps, bounded retries, and a way for an operator to stop the run. A checkpoint cannot undo an external side effect by itself.
How do checkpoints help a run recover?
I store completed steps and their results so a run can resume after an interruption. Recovery also needs repeatable steps, bounded retries, and a way for an operator to stop the run.
Implementation
- Checkpoint after each completed step, including tool results and accumulated state.
- Make every step idempotent, so replaying the boundary step twice is safe.
- Trip a circuit breaker after repeated failure of the same tool rather than retrying indefinitely.
- Distinguish retryable faults from terminal ones and stop on terminal.
- Give the operator a kill path that works from outside the agent's own loop.
What can go wrong
- Message-history checkpoints can omit external side effects.
- Unlimited retries can spend the budget on a tool that cannot recover.
Checks to try
- Kill the worker mid-run and confirm resume produces one consistent result.
- Force a tool to fail repeatedly and confirm the breaker trips.
Questions this pattern answers
How do checkpoints help a run recover?
I store completed steps and their results so a run can resume after an interruption. Recovery also needs repeatable steps, bounded retries, and a way for an operator to stop the run.
What steps does this method use?
Checkpoint after each completed step, including tool results and accumulated state. Make every step idempotent, so replaying the boundary step twice is safe. Trip a circuit breaker after repeated failure of the same tool rather than retrying indefinitely. Distinguish retryable faults from terminal ones and stop on terminal. Give the operator a kill path that works from outside the agent's own loop.
How can I check the result?
Kill the worker mid-run and confirm resume produces one consistent result. Force a tool to fail repeatedly and confirm the breaker trips.
What can go wrong with this approach?
Message-history checkpoints can omit external side effects. Unlimited retries can spend the budget on a tool that cannot recover.
Related projects
ClawKeeper →
ClawKeeper checks proposed Claude agent actions against explicit rules before a model call. Autonomous spend defaults to zero, and the database enforces an append-only audit log.
TechTide Harness Kit →
TechTide Harness Kit uses JSON Schema to declare skill permissions, risk tiers, network access, and approval requirements. Automated checks validate declarations and run prompt-injection fixtures.
CipherClaw →
CipherClaw analyzes recorded agent activity without model calls. It builds causal graphs, scores possible causes, and flags failure patterns.
Swarm 357 →
357 Claude agent roles across six business layers, with portable memory, enforced cost caps, a bash policy gate, and durable checkpoints.
Related patterns
Acceptance tests for agent work →
Executable checks test the task and detect regressions.
Evaluation tests for releases →
A fixed evaluation suite checks release requirements.
MCP tool permissions →
The server checks which tools a caller may use.
Recording agent runs →
A shared trace ID connects the records of a run.
I explain this method here. Current service scope, prices, and terms are on TechTide AI.
Systems Decision Audit at TechTide AI→© 2026 Alex Cinovoj · TechTide AI · Columbus, OH