Polling a batch endpoint every few seconds wastes compute and hides latency, but wiring up webhooks batch inference correctly takes more than a callback URL. You need a contract for payloads, idempotent receivers, and a retry story that survives network partitions. This guide walks the ordered path from submission to reconciliation.
1. Model the batch submission
Generate the job ID client-side. If the gateway assigns it, you must query it before you can correlate the webhook, which defeats the purpose of async delivery. Send a webhook_url in metadata and set a TTL so orphaned jobs expire.
import uuid, requests
job_id = str(uuid.uuid4())
payload = {
"model": "mistral-7b-instruct",
"input": [{"messages": [{"role": "user", "content": q}]} for q in questions],
"metadata": {
"job_id": job_id,
"webhook_url": "https://api.example.com/hooks/batch",
"ttl_seconds": 3600
}
}
requests.post(
"https://gateway.example/v1/batch",
json=payload,
headers={"Authorization": "Bearer KEY"}
)The gateway should acknowledge with 202 and a job_id echo. Treat the local job_id as the primary key for all later state.
2. Define the webhook contract
A webhook is a typed event, not a dump of results. Keep the body small; reference large outputs by URL. Version the event namespace so you can evolve schema without breaking receivers.
{
"event": "batch.completed",
"api_version": "2024-11-01",
"job_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
"status": "success",
"result_url": "https://gateway.example/v1/batch/f47ac10b/results",
"usage": {
"prompt_tokens": 1200,
"completion_tokens": 300
}
}Include terminal events (batch.failed, batch.expired) explicitly. Clients that only listen for completed will hang forever on a degraded provider.
3. Implement idempotent receivers
Delivery is best-effort. You will get the same event twice, sometimes within milliseconds. Persist a dedupe key before any side effect.
from flask import Flask, request, jsonify
import redis
app = Flask(__name__)
r = redis.Redis()
@app.route("/hooks/batch", methods=["POST"])
def batch_hook():
body = request.get_json()
key = f"seen:{body['job_id']}:{body['event']}"
if r.exists(key):
return jsonify({"ok": True}) # already applied
r.setex(key, 86400, "1")
# fetch result_url and write to warehouse
return jsonify({"ok": True})Do not perform the side effect first and dedupe later. A crash between write and dedupe will cause double-processing on retry.
4. Secure the callback
Anyone who learns your endpoint can forge completions. Require an HMAC signature over the raw body, and verify with constant-time comparison.
import hmac, hashlib
from flask import abort
SECRET = b"shared-webhook-secret"
@app.route("/hooks/batch", methods=["POST"])
def batch_hook():
sig = request.headers.get("X-Signature", "")
digest = hmac.new(SECRET, request.get_data(), hashlib.sha256).hexdigest()
if not hmac.compare_digest(sig, digest):
abort(401)
# ... rest of handlerTradeoff: rotating the secret requires dual-write support on the gateway. Agree on a header scheme (X-Signature-Version) before launch. IP allowlisting helps but breaks in multi-tenant gateways that use dynamic egress.
5. Handle retries and backoff
Your handler must return 2xx only after the dedupe key and side effects are committed. If processing takes longer than the gateway’s retry timeout, return 202 Accepted and finish asynchronously, but still guard with the dedupe key.
Common pitfall: returning 500 on a transient database deadlock triggers a tight retry loop that amplifies the outage. Map internal errors to 503 only when you want the gateway to back off; otherwise swallow and ack to avoid duplicate load.
6. Reconcile with polling
Webhooks get dropped. A provider-side bug, a dropped packet, or a receiver outage longer than the retry window means silence. Run a compensating poller on a dead-letter timer.
def reconcile(job_id, timeout=600):
if r.exists(f"seen:{job_id}:batch.completed"):
return
if (time.time() - submitted_at[job_id]) > timeout:
status = requests.get(f"https://gateway.example/v1/batch/{job_id}").json()
if status["status"] == "completed":
process_results(status["result_url"])Keep the poller idempotent against the same dedupe keys. The poller is not a fallback for correctness; it is a safety net for delivery loss.
7. Monitor and meter usage
Capture the usage block from the webhook to attribute cost without an extra API call. An OpenAI-compatible gateway such as n4n.ai meters per-token usage and forwards provider cache-control hints, so the webhook payload is the cheapest place to record spend. Emit a metric per job_id tagged by model and status.
If you batch across multiple providers with automatic fallback, the model field in the webhook may differ from your request. Log both to catch routing surprises.
8. Common pitfalls
Signing with JSON-serialized objects. Serialization order changes between runs. Sign the raw bytes the gateway sent, not a re-parsed dict.
Storing results in the webhook body. A 50 MB batch of completions will exceed most gateway and receiver limits. Use result_url and stream from there with a timeout.
No schema validation. A missing status field should fail closed, not default to success. Validate against your contract with a strict parser.
Mixing auth scopes. Batch submission often uses a long-lived admin key; the result fetch should use a scoped read token. Leaking the admin key to the receiver expands blast radius.
Ignoring clock skew. If you add a timestamp claim to the signature, allow a five-minute window or signature checks will fail during routine NTP steps.
9. Shipping checklist
- Client-generated
job_idin submission - Webhook payload under 4 KB, results by reference
- HMAC verification with versioned secret
- Dedupe key written before side effects
-
2xxonly after commit; async202for slow work - Poller with same dedupe path on 10m timer
- Usage metric emitted per event
Follow that order and webhooks batch inference becomes a boring, reliable part of your pipeline instead of a source of silent data loss.