How to poll vs webhook for long-running LLM tasks

When you submit a batch of 10,000 completions or launch a multi-hour fine-tune, the HTTP response can’t hold the result. You face the classic polling vs webhooks llm tasks decision: repeatedly query a status endpoint, or hand the provider a callback URL and let it push completion events. The right choice changes your infrastructure, error handling, and bill.

Capabilities

What polling gives you

Polling is a client-driven loop. You receive a job ID, then issue GET /jobs/{id} every N seconds until status is succeeded or failed. It works against any API that exposes a job resource, including the OpenAI Batch API and most OpenAI-compatible gateways. The polling vs webhooks llm tasks split shows up in every async LLM API, but polling is the lowest common denominator.

What webhooks give you

Webhooks invert the flow. You register a publicly reachable HTTPS endpoint and pass it in the job request. The server POSTs a signed payload when the job terminates (or at incremental milestones). You never burn cycles asking “are we there yet.” The trade-off is that you now own an internet-facing service.

Price and cost model

Polling is effectively free in token terms—status checks don’t consume inference tokens. However, each poll is an API request. If your gateway enforces per-second request quotas, aggressive polling can starve your real traffic. Per-token usage metering, like that provided by n4n.ai’s OpenAI-compatible endpoint, means the job itself costs the same regardless of notification method; only the overhead differs.

Webhooks shift cost to your side: you need an always-on receiver (or a serverless function) and must handle retries. Providers rarely charge for webhook delivery, but egress from your receiver back to your processing pipeline is yours. If you already run a web service, the marginal cost is near zero.

Latency and throughput

Poll interval sets a hard ceiling on notification latency. Poll every 30s, you’ll learn about completion up to 30s late. Tighten to 2s and you multiply request volume by 15x. For a fleet of 500 concurrent jobs, that’s 250 req/s just for status.

Webhooks deliver within seconds of job finish, often sub-second inside the same cloud region. Throughput on your side is bounded by your receiver’s autoscaling, not by a self-imposed poll loop. For high-job-count pipelines, this removes a major self-inflicted bottleneck.

Ergonomics

Polling fits naturally in a cron job or a worker that already runs. Here’s a minimal Python poll loop against an OpenAI-compatible batch endpoint:

from openai import OpenAI
import time

client = OpenAI(base_url="https://api.example.com/v1")

batch = client.batches.create(
    input_file_id="file-abc",
    endpoint="/v1/embeddings",
    completion_window="24h"
)
job_id = batch.id

while True:
    status = client.batches.retrieve(job_id).status
    if status in ("completed", "failed", "expired"):
        break
    time.sleep(20)

Webhooks require you to stand up an endpoint. With Flask:

from flask import Flask, request, abort
import hmac, hashlib

app = Flask(__name__)
SECRET = b"whsec_xxx"

@app.route("/jobs/callback", methods=["POST"])
def callback():
    sig = request.headers.get("X-Signature", "")
    expected = hmac.new(SECRET, request.data, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(sig, expected):
        abort(401)
    payload = request.json
    if payload["status"] == "completed":
        process_result(payload["output_url"])
    return "", 200

The client side just adds a field:

client.batches.create(
    input_file_id="file-abc",
    endpoint="/v1/embeddings",
    completion_window="24h",
    webhook_url="https://my.app/jobs/callback"
)

Note: the webhook_url extension is common among inference gateways that sit in front of multiple providers; it is not in the vanilla OpenAI SDK, so check your gateway’s schema. Locally, you can test webhooks with a tunnel tool, but polling needs nothing beyond outbound HTTPS.

Ecosystem and routing

Most foundation-model APIs launched with polling because it needs no public ingress. OpenAI’s fine-tuning and batch endpoints are poll-only. Anthropic and Cohere similarly expose job resources. Webhooks appear in event streams (e.g., OpenAI fine-tune events via its dashboard webhooks) but are inconsistent across providers.

An OpenAI-compatible gateway that addresses 240+ models—such as n4n.ai—lets you issue one request shape and rely on automatic fallback when a provider is degraded. Regardless of fallback, your chosen notification method stays client-side; the gateway forwards provider cache-control hints, so a polled retrieve may hit a cached job object, reducing overhead.

Limits and failure modes

Polling limits:

• Status endpoint rate limits (often stricter than inference limits).
• Clock drift or worker crash loses loop state; you must persist job IDs.
• Long polls waste quota if many jobs finish rarely.

Webhook limits:

• Provider retry policy (typically 3–5 attempts with backoff). Miss all, job result is orphaned.
• Your endpoint must validate signatures; a leaked secret allows spoofed completions.
• NAT/firewall rules may block provider IPs; you need stable allowlist.

Head-to-head comparison

Which to choose

Short-lived jobs (<2 min): Poll with a 5–10s interval. The simplicity beats standing up a webhook receiver. If you’re on a serverless function with a hard 15s timeout, just use synchronous calls instead.

High-volume batch (thousands of jobs/day): Webhooks. Polling 1,000 jobs every 20s is 50 req/s of pure overhead. A webhook receiver on a managed function scales to zero and costs cents.

Ephemeral or offline clients: Polling is the only option if your client sits behind NAT or runs in a notebook. Persist job IDs to disk; resume poll after restart.

Regulated environments with no public ingress: Polling from inside a VPC avoids exposing a callback URL. Combine with a gateway that forwards cache-control hints to keep poll overhead low.

Multi-provider routing: If you use a gateway that honors client routing directives and fails over automatically, keep the same notification code. Webhooks survive fallback as long as the gateway rewrites the callback target; polling works regardless because the job ID is stable.

Agentic workflows with long horizons: If a planner spins up subtasks that may run for hours, webhooks prevent a coordinator from busy-waiting. But for a single agent step that waits on one tool call, polling with a short timeout is simpler to debug.

Pick polling when you value simplicity and control; pick webhooks when latency and scale dominate. The polling vs webhooks llm tasks trade-off is not ideological—it’s about where your infrastructure already lives.