How to store agent state in Postgres vs Redis vs S3

When you build multi-step LLM agents, you need a durable place to store agent state postgres redis and S3 each present distinct tradeoffs. Picking the wrong backend turns checkpointing from a safety net into a bottleneck or a silent data-loss bug.

Why checkpointing is a storage problem

Agent loops are stateful. A run accumulates messages, tool outputs, intermediate decisions, and recovery metadata. Checkpointing means writing that state at safe points so a crash, rate limit, or human-in-the-loop pause can resume without replaying expensive model calls. The decision to store agent state postgres redis or in object storage should be driven by access patterns, not hype.

Capabilities

Postgres

Postgres gives you ACID transactions, JSONB columns, and indexing. You can query “all runs stuck at step 3” or join state with usage logs. It supports row-level locking, so concurrent updates to the same run are safe.

Redis

Redis is an in-memory data structure store. It offers strings with TTL, hashes, streams, and pub/sub. It is not a system of record unless you configure AOF persistence and accept recovery lag. Its strength is low-latency reads/writes of small hot objects.

S3

S3 is an object store. Each checkpoint is an immutable blob keyed by path. It supports versioning, lifecycle policies, and server-side encryption. You cannot patch a field inside an object; you rewrite the whole key.

Price and cost model

Postgres cost is dominated by provisioned compute and storage IOPS. Managed services (RDS, Cloud SQL) bill per instance-hour plus storage. Self-hosted shifts cost to ops time.

Redis cost tracks RAM. A 16 GB Redis node stores ~16 GB of state; if you checkpoint 10 KB per run at 1000 runs/sec, memory disappears fast. Elasticache bills per node-hour; memory is the scarce resource.

S3 charges per stored GB-month and per request. Writing 1 million small objects costs fractions of a cent for storage but request costs add up at high checkpoint frequencies. Retrieving via GetObject is cheap; listing prefixes repeatedly is not.

Latency and throughput

Postgres writes land in 1–10 ms on a local network, slower under contention. Connection pooling is mandatory; naive agents that open a connection per step will exhaust max_connections.

Redis operates sub-millisecond for small values on a warmed cache. Throughput scales to 100k+ ops/sec on a single node.

S3 PutObject latency is typically 10–50 ms from a co-located region, but it is not designed for per-step writes in a tight loop. Bulk ingestion is fine; synchronous step-by-step checkpointing is not.

Ergonomics

Writing a checkpoint to Postgres with psycopg2:

import psycopg2
from json import dumps

conn = psycopg2.connect("dbname=agent user=postgres")
cur = conn.cursor()
cur.execute(
    "INSERT INTO agent_state (run_id, step, state) VALUES (%s, %s, %s) "
    "ON CONFLICT (run_id) DO UPDATE SET step=EXCLUDED.step, state=EXCLUDED.state",
    (run_id, step, dumps(state))
)
conn.commit()

Redis is simpler but ephemeral by default:

import redis, json
r = redis.Redis(host="localhost", port=6379)
r.set(f"agent:{run_id}", json.dumps(state), ex=3600)

S3 requires no schema, just a key:

import boto3, json
s3 = boto3.client("s3")
s3.put_object(
    Bucket="agent-state",
    Key=f"{run_id}/step_{step}.json",
    Body=json.dumps(state)
)

The store agent state postgres redis difference shows in reads: Postgres lets you SELECT state FROM agent_state WHERE run_id = %s, Redis needs a single key get, S3 needs a known key or a prefix list.

Ecosystem and tooling

Postgres has migrations (Alembic, Flyway), ORMs (SQLAlchemy), and mature backup tooling. You can run analytical queries over state with window functions.

Redis clients exist for every language; Redis Streams pair well with worker queues if your agent uses a dispatcher.

S3 integrates with Athena, Glue, and lifecycle rules that move old checkpoints to Glacier. If you need to replay runs months later, S3 is the cheapest tombstone.

Limits and failure modes

Postgres fails via connection storms and long-running transactions blocking vacuum. A forgotten BEGIN will quietly degrade everything.

Redis evicts keys under memory pressure unless noeviction is set, silently dropping agent state. AOF rewrite can block; cluster resharding is operational heavy.

S3 provides read-after-write consistency for new objects, but listing prefixes is eventually consistent in some configurations. A deleted prefix may still appear in a list for a short window—bad if you rely on “list to find latest”.

Comparison table

Which to choose

Use Postgres when your agent runs are long-lived, need audit trails, or you must query state across runs. If you store agent state postgres redis together, Postgres is the system of record and Redis is the cache.

Use Redis when checkpoints are hot, short-lived, and loss-tolerant. Session resumes within an hour, and you can rebuild from a deeper store if a key expires. Good for high-frequency loop state where sub-ms matters.

Use S3 when checkpoints are large (full trace dumps, multimodal blobs) or written infrequently. Cold replay, compliance archives, and cross-region replication are natural fits.

Hybrid pattern: Write live state to Redis with a TTL, flush committed checkpoints to Postgres every N steps, and offload raw message logs to S3. This keeps latency low, durability high, and cost bounded.

Pick based on what fails safely for your workload, not on what is trending.