n4nAI
Open console →
API integrationHow-toWebhooks & Async Jobs for Long-Running LLM Tasks

How to retry failed webhook deliveries for LLM jobs

Practical guide for engineers on how to retry failed webhook delivery llm callbacks reliably using idempotency, backoff, and dead-letter queues.

n4n Team4 min read786 words

Audio narration

Coming soon — every post will get a voice note here.

Long-running LLM jobs—batch embeddings, document summarization, fine-tune polls—rarely finish inside a synchronous request. To notify downstream services, you post a callback, and when that POST fails you must retry failed webhook delivery llm results without dropping them or sending duplicates. This guide walks through a production-grade pattern for persisting delivery state, retrying with backoff, and verifying the whole loop works.

Step 1: Define a strict webhook contract and idempotency key

Before writing retry logic, lock down the payload shape and a guarantee that receivers can dedupe. A webhook is just an HTTP POST, but LLM jobs produce large, expensive outputs; you do not want to regenerate or double-deliver them.

Every delivery must carry:

  • job_id: your internal identifier.
  • event: e.g. llm.job.completed.
  • attempt: integer starting at 1.
  • signature: HMAC-SHA256 of the body using a shared secret.
  • Idempotency-Key header: same as job_id plus attempt, so a retried POST is distinguishable but replay-safe.
{
  "job_id": "job_8f2c1a",
  "event": "llm.job.completed",
  "attempt": 1,
  "created_at": "2024-05-12T18:22:01Z",
  "result": {
    "model": "mistral-7b-instruct",
    "tokens": 1820,
    "output": "Summary: ..."
  }
}

The receiver should key its processing on job_id and ignore attempt > 1 if it already persisted that job_id. That turns an at-least-once transport into effectively exactly-once processing.

Step 2: Persist job and delivery state in a durable store

Retries survive process crashes only if state lives in a database, not memory. Use a table that tracks the webhook URL, payload, attempts, and next attempt time.

import sqlalchemy as sa

metadata = sa.MetaData()
webhook_jobs = sa.Table(
    "webhook_jobs", metadata,
    sa.Column("id", sa.String, primary_key=True),
    sa.Column("url", sa.String, nullable=False),
    sa.Column("payload", sa.JSON, nullable=False),
    sa.Column("secret", sa.String, nullable=False),
    sa.Column("attempts", sa.Integer, default=0),
    sa.Column("next_attempt_at", sa.DateTime, nullable=False),
    sa.Column("status", sa.String, default="pending"),  # pending|sent|dead
)

When an LLM job finishes, insert a row with next_attempt_at = now(). A separate worker claims rows where next_attempt_at <= now() and status = 'pending'.

To avoid two workers sending the same webhook, claim atomically:

with engine.begin() as conn:
    job = conn.execute(
        sa.text("""
          UPDATE webhook_jobs
          SET status = 'claimed', attempts = attempts + 1
          WHERE id = (
            SELECT id FROM webhook_jobs
            WHERE status = 'pending' AND next_attempt_at <= :now
            ORDER BY next_attempt_at
            LIMIT 1 FOR UPDATE SKIP LOCKED
          )
          RETURNING id, url, payload, secret, attempts
        """),
        {"now": datetime.utcnow()},
    ).fetchone()

SKIP LOCKED is available in Postgres; on SQLite use a single-threaded worker or a busy flag with timeout.

Step 3: Dispatch the LLM job and register the callback

Assume you run the LLM call via an OpenAI-compatible client. The moment the result is ready, you store the webhook row. Do not block the LLM response on the callback POST.

from openai import OpenAI

client = OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")

def run_llm_job(job_id: str, prompt: str, webhook_url: str, secret: str):
    resp = client.chat.completions.create(
        model="mistral-7b-instruct",
        messages=[{"role": "user", "content": prompt}],
        timeout=120,
    )
    payload = {
        "job_id": job_id,
        "event": "llm.job.completed",
        "created_at": datetime.utcnow().isoformat(),
        "result": {"output": resp.choices[0].message.content,
                   "tokens": resp.usage.total_tokens},
    }
    with engine.begin() as conn:
        conn.execute(webhook_jobs.insert().values(
            id=job_id, url=webhook_url, payload=payload, secret=secret,
            attempts=0, next_attempt_at=datetime.utcnow(), status="pending"))

This decouples inference from delivery. If the webhook later fails, you still have the payload and can retry failed webhook delivery llm callbacks independently.

Step 4: Implement the delivery function with hard timeouts

A hung TCP connection is the silent killer. Always set a connect and read timeout, and treat any non-2xx as failure.

import hmac, hashlib, requests

def sign_body(secret: str, body: bytes) -> str:
    return hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()

def send_webhook(url: str, payload: dict, secret: str, attempt: int):
    body = json.dumps(payload).encode()
    headers = {
        "Content-Type": "application/json",
        "Idempotency-Key": f"{payload['job_id']}:{attempt}",
        "X-Signature": sign_body(secret, body),
    }
    try:
        r = requests.post(url, data=body, headers=headers, timeout=(3, 10))
    except requests.RequestException as e:
        raise RuntimeError(f"network error: {e}")
    if not 200 <= r.status_code < 300:
        raise RuntimeError(f"bad status {r.status_code}")

Note the tuple timeout: 3 seconds to connect, 10 to read. Tune based on your receiver’s p99.

Step 5: Schedule retries with exponential backoff and jitter

Naive fixed intervals stampede the receiver when it recovers. Use exponential backoff with full jitter and a max attempt cap (typically 5–7).

import random, math, datetime

MAX_ATTEMPTS = 6
BASE_DELAY = 30  # seconds

def compute_next_attempt(attempt: int) -> datetime:
    delay = min(BASE_DELAY * (2 ** (attempt - 1)), 3600)
    jitter = random.uniform(0, delay)
    return datetime.utcnow() + datetime.timedelta(seconds=jitter)

def process_pending():
    job = claim_job()  # from Step 2
    if not job:
        return
    id_, url, payload, secret, attempts = job
    try:
        send_webhook(url, payload, secret, attempts)
        with engine.begin() as conn:
            conn.execute(webhook_jobs.update().where(webhook_jobs.c.id == id_)
                         .values(status="sent"))
    except Exception as e:
        log.warning("delivery failed: %s", e)
        if attempts >= MAX_ATTEMPTS:
            with engine.begin() as conn:
                conn.execute(webhook_jobs.update().where(webhook_jobs.c.id == id_)
                             .values(status="dead"))
            alert_dead_letter(id_)
        else:
            next_at = compute_next_attempt(attempts)
            with engine.begin() as conn:
                conn.execute(webhook_jobs.update().where(webhook_jobs.c.id == id_)
                             .values(status="pending", next_attempt_at=next_at))

Run process_pending on a cron or a long-loop worker every 10–30 seconds. The jitter spreads retries so a fleet of failed jobs doesn’t retry in lockstep.

Step 6: Add a dead-letter queue and observability

When max attempts hit, the row moves to status='dead'. That is not “ignore.” Ship those IDs to a monitoring system (Sentry, CloudWatch, or a simple Slack webhook) and keep the payload for manual replay.

def alert_dead_letter(job_id: str):
    requests.post(SLACK_URL, json={"text": f"Dead webhook: {job_id}"}, timeout=5)

Also emit metrics: webhook_delivery_attempts, webhook_delivery_success, webhook_delivery_dead. A sudden spike in dead letters means the receiver is down or your signature header changed.

Step 7: Verify successful delivery end-to-end

You cannot claim reliability without a test that forces a failure. Use a local receiver that returns 500 for the first two hits, then 200.

from flask import Flask, request, jsonify

app = Flask(__name__)
state = {"hits": 0}

@app.route("/hook", methods=["POST"])
def hook():
    state["hits"] += 1
    if state["hits"] <= 2:
        return jsonify({"error": "simulated failure"}), 500
    # verify signature
    sig = request.headers.get("X-Signature")
    expected = hmac.new(SECRET.encode(), request.get_data(), hashlib.sha256).hexdigest()
    assert hmac.compare_digest(sig, expected), "bad sig"
    return jsonify({"ok": True}), 200

Start it on port 5000, point your worker’s webhook_url to http://localhost:5000/hook, and insert a fake job:

python worker.py &  # runs process_pending loop
curl -X POST localhost:5000/internal_test -d '{"prompt":"test","webhook_url":"http://localhost:5000/hook"}'

Watch logs: you should see two failures, then a success on the third attempt (attempt=3). Confirm the receiver’s state["hits"] equals 3 and the webhook_jobs row is sent.

For signature verification on the receiver side, always use constant-time compare. For the sender, rotate secrets per job batch if you can; a leaked secret lets attackers forge completions.

Edge cases that will bite you

Clock skew. If your next_attempt_at uses UTC but the worker box is in local time, jobs fire early or late. Always store and compare UTC.

Large payloads. LLM outputs can be tens of KB. Some receivers reject bodies >1MB. If your result is huge, deliver a signed URL to the result object instead of the full text in the webhook.

Duplicate LLM jobs. If the producer crashes after inserting the webhook row but before marking the LLM job done, you may get two rows for one logical job. Make job_id the primary key and use INSERT ... ON CONFLICT DO NOTHING.

Receiver returns 200 but fails internally. The webhook contract should require the receiver to verify signature before ack. If they 200 early and then crash, you’ve lost the data. Consider a two-phase approach: they return 202 Accepted, then call your status endpoint. That’s heavier; for most teams, a strict idempotency key is enough.

To retry failed webhook delivery llm events at scale, the pattern above—durable state, atomic claim, signed payload, backoff with jitter, dead-letter—is the difference between “looks fine in dev” and “survives a provider outage.” Build the test harness before you trust it in production.

Tagswebhooksretryreliabilityasync

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationDesigning idempotent webhook handlers for LLM job callbacksPractical guide to building an idempotent webhook handler llm callback system: dedupe strategies, concurrency control, replay testing, and pitfalls.API integrationBuilding a webhook receiver for async LLM completionsStep-by-step tutorial for building a secure webhook receiver for async LLM completions using Python and FastAPI, with runnable code and verification.API integrationWebhooks for batch inference: what to know before buildingBuild reliable webhooks batch inference: define contracts, sign callbacks, handle retries idempotently, and reconcile with polling to avoid LLM pipeline gaps.API integrationHow to poll vs webhook for long-running LLM tasksA head-to-head guide on polling vs webhooks for long-running LLM tasks covering cost, latency, ergonomics, and limits, plus a use-case verdict.API integrationUsing webhooks for long-running LLM batch jobsLearn how to implement webhooks for long-running LLM batch jobs: upload batches, watch status, and receive completion callbacks with signed HTTP posts.API integrationHandling long-running LLM tasks without blocking requestsLearn how to handle long running LLM tasks asynchronously using job queues and webhooks to keep your API responsive and resilient to provider latency.

All webhooks & async jobs for long-running llm tasks posts →