Recovery and durable checkpoints
Long runs fail for boring reasons: a deploy, an OOM, a provider timeout. Checkpointing makes those events cheap.
What is durable checkpoints and recovery for agent runs?
Long runs fail for boring reasons: a deploy, an OOM, a provider timeout. Checkpointing makes those events cheap.
Implementation
- Checkpoint after each completed step, including tool results and accumulated state.
- Make every step idempotent, so replaying the boundary step twice is safe.
- Trip a circuit breaker after repeated failure of the same tool rather than retrying indefinitely.
- Distinguish retryable faults from terminal ones and stop on terminal.
- Give the operator a kill path that works from outside the agent's own loop.
Failure modes
- Checkpointing only the message history, which loses external side effects.
- Infinite retry loops that burn budget on a permanently failing tool.
How it is verified
- Kill the worker mid-run and confirm resume produces one consistent result.
- Force a tool to fail repeatedly and confirm the breaker trips.
Questions this pattern answers
What is durable checkpoints and recovery for agent runs?
Long runs fail for boring reasons: a deploy, an OOM, a provider timeout. Checkpointing makes those events cheap.
How do you implement recovery and durable checkpoints?
Checkpoint after each completed step, including tool results and accumulated state. Make every step idempotent, so replaying the boundary step twice is safe. Trip a circuit breaker after repeated failure of the same tool rather than retrying indefinitely. Distinguish retryable faults from terminal ones and stop on terminal. Give the operator a kill path that works from outside the agent's own loop.
How is recovery and durable checkpoints verified?
Kill the worker mid-run and confirm resume produces one consistent result. Force a tool to fail repeatedly and confirm the breaker trips.
What usually goes wrong with recovery and durable checkpoints?
Checkpointing only the message history, which loses external side effects. Infinite retry loops that burn budget on a permanently failing tool.
Projects in this track
ClawKeeper →
Governed Claude agent control plane. A deterministic policy engine runs before every LLM call, autonomous spend defaults to zero, and the audit log is enforced by the database.
TechTide Harness Kit →
Trust contracts for agent skills expressed as JSON Schema: risk tiers, tool permissions, deny-by-default egress, approval gates, and prompt-injection fixtures behind a wall of CI validation gates.
CipherClaw →
Deterministic multi-agent debugger with zero LLM calls in the analysis path: causal graphs, weighted root-cause scoring, and failure prediction.
Swarm 357 →
357 Claude agent roles across six business layers, with portable memory, enforced cost caps, a bash policy gate, and durable checkpoints.
Related patterns
This page documents how the pattern is built. Engagement scope and pricing live on TechTide AI.
AI Production Systems at TechTide AI→© 2026 Alex Cinovoj · TechTide AI · Columbus, OH