Long-running LLM inference jobs abandon the synchronous request-response model. A robust idempotent webhook handler llm callback endpoint must absorb duplicate deliveries, out-of-order events, and partial provider outages without double-processing results or corrupting application state.
1. Map the job state machine first
Before writing a single route, define the lifecycle of an async LLM task. Most gateways emit at least job.created, job.completed, and job.failed. Some send job.progress increments, and a few emit job.expired if a task times out in the queue.
Treat the webhook as a notification, not the source of truth. Your job record lives in your database; the callback only advances it. If you let the callback body become the authoritative state, a replayed older event can overwrite a newer one.
{
"event_id": "evt_01H9X8K",
"job_id": "job_8a7c3f",
"type": "job.completed",
"created_at": 1715270000,
"result": {
"model": "mistral-7b",
"choices": [{"text": "The capital of France is Paris."}]
}
}Pitfall: assuming created_at is monotonic across providers. Clocks drift, and a retry can carry a stale timestamp. Use event_id for dedupe, not time.
2. Derive a stable idempotency key
The key must be unique per delivered event, not per job. A single job can emit many events: queued, progress, completed, failed. Concatenate job_id and event_id, or store event_id alone if the provider guarantees global uniqueness (most do).
def idempotency_key(payload: dict) -> str:
return f"{payload['job_id']}:{payload['event_id']}"Store processed keys in a table with a unique constraint. This is the core of any idempotent webhook handler llm integration, and it is cheaper than a general-purpose distributed log.
CREATE TABLE webhook_events (
key TEXT PRIMARY KEY,
job_id TEXT NOT NULL,
type TEXT NOT NULL,
received_at TIMESTAMPTZ DEFAULT now()
);If you operate in a multi-region setup, consider a UUIDv7 or ULID for event_id from the provider; if not, the string key above is sufficient.
3. Persist before you act
Inside the handler, wrap the dedupe check and state transition in one database transaction. If the key already exists, return 200 immediately and skip side effects. Do not perform external calls inside the transaction.
import psycopg2
def handle_webhook(payload):
key = f"{payload['job_id']}:{payload['event_id']}"
conn = psycopg2.connect(DATABASE_URL)
with conn:
with conn.cursor() as cur:
cur.execute(
"INSERT INTO webhook_events (key, job_id, type) VALUES (%s, %s, %s) "
"ON CONFLICT (key) DO NOTHING RETURNING key;",
(key, payload['job_id'], payload['type'])
)
if cur.fetchone() is None:
return {"status": "duplicate"} # already processed
cur.execute(
"UPDATE jobs SET status = %s, updated_at = now() WHERE id = %s",
(payload['type'], payload['job_id'])
)
# side effects after commit
dispatch_result_to_user(payload)
return {"status": "ok"}Tradeoff: holding a transaction during side effects risks lock contention and connection exhaustion. Perform only the state mutation in the DB; do external calls after commit, but record a processed_at flag so a crash before side-effect completion can be retried safely.
Use READ COMMITTED isolation; the unique constraint handles the race. SERIALIZABLE adds unnecessary latency here.
4. Survive concurrent redeliveries
Providers retry on network errors or non-2xx responses. Two identical deliveries can hit different workers within milliseconds. The unique constraint on webhook_events.key resolves the race: one INSERT wins, the other gets a conflict and short-circuits.
If your side effect is not DB-backed (e.g., publishing to Kafka or calling a third-party API), use the same key as the message key to preserve per-job ordering, or employ a short-lived distributed lock:
from redis import Redis
r = Redis()
def claim_lock(key):
return r.set(key, "1", nx=True, ex=30)Release after the side effect completes. If the process dies, the lock expires and a later retry can proceed. This pattern complements the DB dedupe rather than replacing it.
5. Guard downstream side effects
An idempotent webhook handler llm receiver often triggers emails, database writes, or billing adjustments. Pass the event key to those systems as their idempotency token. Stripe-style Idempotency-Key headers work well for HTTP calls.
curl -X POST https://api.internal/notify \
-H "Idempotency-Key: job_8a7c3f:evt_01H9X8K" \
-d '{"job_id":"job_8a7c3f","result":"ok"}'If the downstream lacks idempotency support, write a local outbox table and mark rows sent only after ACK. Replays then skip unsent rows. Exactly-once delivery is impossible over HTTP; aim for effectively-once by making every write keyed.
6. Authenticate every callback
Never process unverified bodies. Most gateways sign payloads with HMAC-SHA256 using a shared secret. Verify before parsing JSON.
import crypto from 'crypto';
function verifySig(rawBody: string, sigHeader: string, secret: string): boolean {
const expected = crypto.createHmac('sha256', secret).update(rawBody).digest('hex');
return crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(sigHeader));
}Pitfall: verifying after JSON.parse on a different byte sequence than signed. Always hash the raw request bytes from the body stream. Reject with 401 if mismatch; do not fall back to processing.
7. Return correct status codes
A 2xx tells the sender to stop retrying. Return 500 (or 503) if your DB is down or side effect failed, so the provider retries. But if you detected a duplicate, return 200—otherwise you’ll get stuck in a retry loop for an event you already applied.
if duplicate:
return ("", 200)For poison payloads (malformed, invalid signature, unknown job), return 400 to stop retries and log for manual review. A dead-letter queue should capture these for forensics.
8. Reconcile with polling
Webhooks get lost, especially during provider incidents. Build a watchdog that polls the job status endpoint every minute for jobs stuck in processing beyond expected latency. This compensates for missed callbacks and makes your idempotent webhook handler llm pipeline resilient to silent drops.
def reconcile():
stuck = query(
"SELECT id FROM jobs WHERE status='processing' "
"AND updated_at < now() - interval '5 min'"
)
for job_id in stuck:
status = gateway.get_job(job_id)
if status in ("completed", "failed"):
handle_webhook(build_synthetic_event(job_id, status))Synthetic events must use the same key scheme (e.g., job_id:synthetic_<status>) so they dedupe against real ones. Scale the watchdog by sharding job IDs across workers if you process millions of tasks.
9. Test with replay and shuffle
Write an integration test that fires the same job.completed event ten times, interleaved with job.progress and a late job.failed. Assert the job status is terminal and side effects fired exactly once.
for i in $(seq 1 10); do
curl -X POST localhost:8000/webhook -d @sample_event.json
sleep 0.$RANDOM
doneAdd random delays to simulate out-of-order delivery. If your handler mutates state on job.failed after job.completed, the test will catch it. Run this in CI against a throwaway Postgres and Redis.
Common pitfalls we keep seeing
- Using
job_idalone as dedupe key: progress and completion collide, causing lost updates. - Writing state before inserting the event key, causing partial applies on retry.
- Returning 200 on internal failure, silencing retries and hiding data loss.
- Trusting
created_atfor ordering instead of explicit sequence numbers or event-type precedence. - Skipping signature verification because “it’s internal network” — lateral movement is real.
An idempotent webhook handler llm design is not glamorous, but it is the difference between a demo that works on a sunny day and a system that survives a provider incident without double-charging a customer or sending the same summary email nine times.