Most agent frameworks treat a crash as game over: the context vanishes and the task restarts from zero. Building resumable AI agents means persisting state at granular checkpoints so a process can wake up exactly where it left off, even after a SIGKILL or a cloud instance reclaim.
Prerequisites
- Python 3.11+ with
pip install redis openai - A local Redis instance (
redis-serveron:6379) - An OpenAI-compatible LLM endpoint. If you want automatic fallback when a provider is rate-limited, point the client at a gateway like n4n.ai that exposes one OpenAI-compatible endpoint for 240+ models and routes around degraded providers.
You should already understand basic agent loops: send messages, get completions, call tools, repeat.
Why most frameworks drop the ball
They keep state in process memory and treat the LLM call as a black box. The moment the container dies, the in-memory message list is gone. Recovery means re-prompting from the user goal, which wastes tokens and breaks half-finished tool side effects. Resumable AI agents demand that state lives outside the process and that every mutation is replayable.
Designing the checkpoint schema
A checkpoint is a frozen snapshot of everything needed to continue. Skip nothing: conversation history, current step index, pending tool calls, and completed actions.
from dataclasses import dataclass, field, asdict
import json
import time
@dataclass
class AgentState:
task_id: str
step: int
messages: list[dict] = field(default_factory=list)
pending_tool: str | None = None
completed_tools: list[str] = field(default_factory=list)
status: str = "running" # running | done | failed
updated_at: float = field(default_factory=time.time)
def to_json(self) -> str:
return json.dumps(asdict(self))
@classmethod
def from_json(cls, raw: str) -> "AgentState":
return cls(**json.loads(raw))The completed_tools list is what makes tool execution idempotent later.
Writing the checkpoint sink
Write to Redis atomically with a namespaced key and a TTL so orphaned tasks expire.
import redis
r = redis.Redis(host="localhost", port=6379, db=0)
def save_checkpoint(state: AgentState) -> None:
key = f"agent:checkpoint:{state.task_id}"
r.set(key, state.to_json(), ex=86400)
print(f"[checkpoint] saved step {state.step} for {state.task_id}")Building the agent loop with yield points
The loop saves state before any side-effecting operation. If the process dies, the last checkpoint is the recovery point. We simulate a crash on step 2 to prove the pattern.
from openai import OpenAI
import os
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
def run_agent(task_id: str, user_goal: str, resume: bool = False) -> None:
if resume:
raw = r.get(f"agent:checkpoint:{task_id}")
if raw:
state = AgentState.from_json(raw)
print(f"[resume] restored at step {state.step}")
else:
print("[resume] no checkpoint found, starting fresh")
state = AgentState(task_id=task_id, step=0,
messages=[{"role": "user", "content": user_goal}])
else:
state = AgentState(task_id=task_id, step=0,
messages=[{"role": "user", "content": user_goal}])
while state.status == "running":
save_checkpoint(state) # yield point before LLM call
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=state.messages,
)
msg = resp.choices[0].message.model_dump()
state.messages.append(msg)
state.step += 1
if not msg.get("tool_calls"):
state.status = "done"
save_checkpoint(state)
print("[agent] task complete")
return
tool_call = msg["tool_calls"][0]
tool_id = tool_call["id"]
if tool_id in state.completed_tools:
continue # already ran before a crash
state.pending_tool = tool_call["function"]["name"]
save_checkpoint(state)
# simulate executing the tool
if state.step == 2:
print("[crash] simulating mid-task failure")
os._exit(1)
state.completed_tools.append(tool_id)
state.pending_tool = None
state.messages.append({
"role": "tool",
"tool_call_id": tool_id,
"content": f"executed {tool_call['function']['name']}"
})Running the crash demo
Execute without resume:
python agent.py --task t1 --goal "Book a flight to NYC"Expected output:
[checkpoint] saved step 0 for t1
[checkpoint] saved step 1 for t1
[crash] simulating mid-task failureThe process exits non-zero. Redis still holds the step-1 checkpoint with pending_tool set.
Resuming after a kill
Rerun with the resume flag. The loader pulls the last state and continues without re-running step 0.
python agent.py --task t1 --goal "Book a flight to NYC" --resumeOutput:
[resume] restored at step 1
[checkpoint] saved step 1 for t1
[checkpoint] saved step 2 for t1
[agent] task completeThe resumable AI agents pattern turned a fatal crash into a two-line restart. No tokens wasted on replaying the user goal from scratch.
Scaling checkpoints with write-ahead logs
Serializing the full message list every step is fine for short tasks but blows up memory and Redis payloads on long runs. Use an append-only log and rebuild state by replay:
def append_event(task_id: str, event: dict) -> None:
r.rpush(f"agent:log:{task_id}", json.dumps(event))
def replay_log(task_id: str) -> AgentState:
raw_events = r.lrange(f"agent:log:{task_id}", 0, -1)
state = AgentState(task_id=task_id, step=0)
for e in raw_events:
ev = json.loads(e)
# apply event to state (add message, mark tool done, etc.)
if ev["type"] == "message":
state.messages.append(ev["data"])
elif ev["type"] == "tool_done":
state.completed_tools.append(ev["tool_id"])
state.step += 1
return stateEach checkpoint becomes O(1) and corruption is isolated to a single event.
Dealing with LLM provider outages
A crash isn’t the only mid-task hazard. Providers throw 429s or time out. If your client talks to a single vendor, the agent stalls. For example, n4n.ai provides one OpenAI-compatible endpoint that automatically falls back when a provider is rate-limited or degraded, which complements checkpointing: the agent loop stays alive, and if it still dies, the checkpoint restores it. The same resilience mindset applies—assume the dependency will fail and design for continuation.
Idempotent tool execution
When you resume, any tool that already executed before the crash must not double-execute. The completed_tools list in AgentState already stamps each tool call ID. On resume, the loop skips known IDs. This converts “at least once” delivery into “exactly once” for side effects like booking or payments.
if tool_id in state.completed_tools:
state.pending_tool = None
continueAdd a short TTL on the tool side if the external system doesn’t support idempotency keys natively.
Production checklist for resumable AI agents
- Persist state before every external call (LLM, tool, network).
- Store enough to rebuild memory, pending actions, and completed work.
- Use a durable store with TTLs for cleanup.
- Replay append-only logs instead of full snapshots for long runs.
- Guard every tool with idempotency keys or completed-action records.
- Test by sending
SIGKILLduring a run, then resuming.
Ship with these and a node restart becomes a non-event rather than a user-visible failure.