Designing agent workflows that survive a server restart

Most agent code dies the moment its process dies. If you want durable agent workflows that actually run for hours or days, you need to stop writing linear scripts and start persisting execution state after every meaningful step. This guide lays out an ordered path to make your agents restart-safe, from state modeling to chaos testing.

1. Model the agent as an explicit state machine

A durable agent workflow is a state machine, not a Python function that calls await in a loop. Define the states up front. Each transition corresponds to a step that either succeeds, fails, or times out.

from enum import Enum

class Step(Enum):
    START = "start"
    CALL_LLM = "call_llm"
    PARSE_RESPONSE = "parse"
    INVOKE_TOOL = "tool"
    HUMAN_REVIEW = "review"
    DONE = "done"
    ERROR = "error"

Persist the current step and the associated payload in a store you control—SQLite, Postgres, or Redis. Do not keep this in process memory. When the server restarts, the first thing your worker does is read the row and resume from Step.CALL_LLM instead of Step.START.

The pitfall here is implicit state: a variable like retries = 3 living on the stack. If the box reboots, that context is gone. Make every piece of state you care about serializable and written down before you act on it.

2. Checkpoint before and after each side effect

A side effect is any call that changes the world: an LLM completion, a database write, an outbound webhook, a payment. For durable agent workflows, you checkpoint twice—once with intent, once with result.

import sqlite3, json

def save_checkpoint(conn, run_id, step, data):
    conn.execute(
        "INSERT INTO checkpoints(run_id, step, data, ts) VALUES (?,?,?,datetime('now'))",
        (run_id, step, json.dumps(data))
    )
    conn.commit()

def latest_checkpoint(conn, run_id):
    row = conn.execute(
        "SELECT step, data FROM checkpoints WHERE run_id=? ORDER BY ts DESC LIMIT 1",
        (run_id,)
    ).fetchone()
    return (row[0], json.loads(row[1])) if row else None

Before calling the LLM, write {step: "call_llm", prompt: ...}. After the response arrives, write {step: "call_llm", response: ..., token_usage: ...}. If the process is killed between the two writes, on restart you see the intent checkpoint and know you must re-issue the call (or check for a cached result).

The tradeoff is write amplification. For high-throughput agents, batch checkpoints or use an append-only log with periodic snapshots. But never skip the post-action write; that is where most silent corruption comes from.

3. Enforce idempotency on every action

Restart recovery will replay steps. If your tool call sends an email, a crash mid-flight can send it twice. Wrap side effects with a deterministic key and a done-marker table.

def execute_once(conn, key, fn):
    if conn.execute("SELECT 1 FROM executed WHERE key=?", (key,)).fetchone():
        return
    fn()
    conn.execute("INSERT INTO executed(key) VALUES (?)", (key,))
    conn.commit()

Use keys derived from the run and step: f"{run_id}:send_slack:42". For external APIs that support idempotency headers (Stripe, some LLM gateways), pass the run ID as the idempotency key. For APIs that do not, you must reconcile after the fact—query whether the object was created.

A common mistake is assuming LLM outputs are deterministic. They are not. Idempotency applies to your actions, not to the model. Store the prompt hash and the response ID so you can detect replays without regenerating.

4. Drive execution from a durable queue, not a memory loop

Do not write while True: run_agent() and hope the OS keeps you alive. On boot, query for all runs that are not in a terminal state and enqueue them.

def recover_runs(conn):
    rows = conn.execute(
        "SELECT DISTINCT run_id FROM checkpoints "
        "WHERE step NOT IN ('done','error')"
    ).fetchall()
    for (run_id,) in rows:
        task_queue.put(run_id)

This decouples scheduling from execution. A worker can die; another picks up the queue. For durable agent workflows at scale, use a real queue (SQS, RabbitMQ, or a DB table with SKIP LOCKED), but the principle is identical: the source of truth is the store, not the worker’s call stack.

5. Treat LLM calls as retryable network calls

Model providers rate-limit, return 529, or silently truncate. Build retries with backoff, and design your checkpointing so a retried call does not duplicate state transitions.

An inference gateway such as n4n.ai fronts 240+ models behind one OpenAI-compatible endpoint and automatically fails over when a provider is rate-limited or degraded, but your code must still store the returned completion ID and checkpoint the output before moving on.

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="...")

def call_llm(run_id, prompt, model="gpt-4o-mini"):
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        metadata={"run_id": run_id}  # forwarded as routing hint if supported
    )
    return resp

Wrap this in the checkpoint pattern from step 2. If the request throws, the intent checkpoint lets you retry safely. If it succeeds, the result checkpoint ensures you never pay for the same token stream twice.

6. Externalize delays and human pauses

Agents often need to wait: for a job to finish, for a human to approve, for rate limits to reset. Never time.sleep(3600) inside the worker. Store a resume_at timestamp and let the scheduler wake the run later.

SELECT run_id FROM checkpoints
WHERE step = 'review'
  AND resume_at <= datetime('now')
  AND status = 'pending';

This keeps workers free and makes crashes irrelevant—the clock lives in the database. The tradeoff is latency: you need a ticker (every 10–30 seconds) to poll, or use LISTEN/NOTIFY in Postgres for immediate wake-ups.

7. Crash-test with kill -9

Theory is cheap. Run a long agent, then kill -9 the worker mid-step. Restart the process. Verify the run completes with exactly-once side effects.

Write a test that:

1. Starts a run that sends a fake email at step 3.
2. Kills the worker after the intent checkpoint but before the send.
3. Restarts and asserts one email, not two.

If you cannot kill your own agent and have it finish correctly, your durable agent workflows are not durable—they are lucky.

8. Emit recovery-friendly logs

Structured logs are the difference between a 2 a.m. page you can solve and one you cannot. Every log line must carry run_id, step, and checkpoint_id.

{"run_id":"r-8821","step":"call_llm","checkpoint_id":119,"msg":"retry 2","tokens":412}

When a run stalls, you open the log, find the last checkpoint, and know exactly what the worker thought it was doing. Avoid logging full prompts at info level; that bloats storage and leaks data. Log hashes.

9. Tradeoffs: when not to build this

Checkpointing, queues, and idempotency wrappers add real code. If your agent finishes in under 60 seconds and has no costly side effects, a simple retry-from-start wrapper is fine. Durable agent workflows earn their complexity only when the cost of a restart is higher than the cost of the infrastructure.

Conversely, if you are orchestrating multi-hour research agents, invoice generation, or anything touching money, the patterns above are not optional. The ordered path is: state machine → double checkpoint → idempotent actions → queue-driven recovery → retryable LLM layer → externalized waits → kill-test → logs.

Ship the smallest version of this that survives a power cycle. Then expand.