Handling long-running LLM tasks without blocking requests

Synchronous HTTP calls to language models fall over the moment a prompt needs deep reasoning, long context, or batch generation. To handle long running llm tasks without tying up your web servers, you need to decouple request acceptance from inference execution using a job queue and webhooks. This guide walks through a concrete pattern you can ship today.

Step 1: Identify which requests must go async

Not every LLM call needs a queue. Set a hard latency budget for your synchronous path—typically under 5 seconds for user-facing endpoints. Anything that may exceed that, or that uses more than ~4K output tokens, should be offloaded.

Instrument your existing calls to log duration and token counts. A simple middleware snippet in FastAPI shows where the pain is:

import time
from fastapi import Request
import logging

@app.middleware("http")
async def log_llm_latency(request: Request, call_next):
    start = time.monotonic()
    response = await call_next(request)
    dur = time.monotonic() - start
    if request.url.path.startswith("/v1/generate"):
        logging.info({"path": request.url.path, "ms": dur*1000})
    return response

If logs show tail latency above your budget, those routes are candidates to handle long running llm tasks via async jobs. Don’t guess—measure. A 2-second median with a 30-second p99 is a clear signal that the synchronous design is broken.

Also consider batch endpoints. If a client submits 50 summaries at once, running them inline will starve every other request on the same worker. Queue them.

Step 2: Stand up a minimal job queue

Redis plus RQ is the fastest way to get a durable queue without Kubernetes operators or heavyweight brokers. Install the deps:

pip install redis rq fastapi uvicorn requests

Start Redis locally:

docker run -d -p 6379:6379 redis:7

Define a queue connection module so both API and worker share the same handle:

# queue.py
import redis
from rq import Queue

conn = redis.Redis(host="localhost", port=6379, db=0)
llm_queue = Queue("llm_jobs", connection=conn)

This gives you a named queue where API processes enqueue and separate worker processes consume. You can later add priority queues (high, low) by declaring more Queue objects. The key point: the web tier never imports the model client.

Step 3: Submit the LLM job from the API layer

Your HTTP handler should validate input, enqueue a job, and return 202 Accepted with a job ID. Never call the model inside the request loop.

# api.py
from fastapi import FastAPI
from queue import llm_queue
import uuid

app = FastAPI()

@app.post("/v1/async-generate")
async def async_generate(payload: dict):
    job_id = str(uuid.uuid4())
    llm_queue.enqueue(
        "worker.run_llm_job",
        job_id,
        payload,
        webhook_url=payload.get("callback")
    )
    return {"job_id": job_id, "status": "queued"}, 202

The client gets an immediate response. The callback field is a URL you will POST results to when done. If the client cannot receive webhooks, it can poll /v1/jobs/{job_id} later. This split is the core of how you handle long running llm tasks without blocking the request thread.

Step 4: Run inference in a worker with a resilient client

Write a worker script that pulls jobs and calls the model. Use the OpenAI Python client pointed at an OpenAI-compatible gateway. If you point it at n4n.ai’s OpenAI-compatible endpoint, you get automatic fallback when a provider is rate-limited or degraded, so a slow backend doesn’t stall your job or require you to write custom retry logic.

# worker.py
import os, json, requests
from openai import OpenAI
from rq import Worker, Queue, Connection
import redis

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # OpenAI-compatible, 240+ models
    api_key=os.environ["N4N_API_KEY"]
)

def run_llm_job(job_id: str, payload: dict, webhook_url: str = None):
    model = payload.get("model", "gpt-4o-mini")
    resp = client.chat.completions.create(
        model=model,
        messages=payload["messages"],
        max_tokens=payload.get("max_tokens", 2000),
        timeout=120  # worker-level guard
    )
    result = {
        "job_id": job_id,
        "content": resp.choices[0].message.content,
        "usage": resp.usage.model_dump()
    }
    if webhook_url:
        requests.post(webhook_url, json=result, timeout=10)
    else:
        r = redis.Redis()
        r.set(f"job:{job_id}", json.dumps(result))
    return result

if __name__ == "__main__":
    with Connection(redis.Redis()):
        Worker([Queue("llm_jobs")]).work()

The timeout on the client prevents a single hung request from blocking the worker indefinitely. The gateway’s fallback means you don’t write your own retry loop across providers. For longer jobs, bump the client timeout and the RQ result_ttl so the job record isn’t evicted before completion.

Step 5: Deliver results via webhook or poll

Webhooks push results; polling pulls. Provide both for robustness. Add a status endpoint to the API:

# api.py (extend)
from redis import Redis
import json

r = Redis()

@app.get("/v1/jobs/{job_id}")
async def get_job(job_id: str):
    data = r.get(f"job:{job_id}")
    if not data:
        return {"status": "pending"}, 202
    return json.loads(data)

In the worker, if no webhook is supplied, write to Redis as shown above. Clients can poll every few seconds. For webhook receivers, validate the payload signature if you add one; at minimum check job_id against your DB to avoid accepting spoofed completions. A well-designed system to handle long running llm tasks treats the webhook as untrusted input.

If you expect huge result payloads (e.g., 20K tokens), don’t inline them in the webhook. Return a signed URL or store in object storage and send the reference.

Step 6: Verify the pipeline end to end

Start the worker in one terminal:

python worker.py

Start the API:

uvicorn api:app --port 8000

Trigger a job with a long prompt:

curl -X POST localhost:8000/v1/async-generate \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role":"user","content":"Write a 2000-word technical essay on distributed caches."}],
    "max_tokens": 2000,
    "callback": "https://webhook.site/your-unique-url"
  }'

You should immediately get:

{"job_id":"<uuid>","status":"queued"}

Within a minute (depending on model speed), your webhook receives:

{
  "job_id":"<uuid>",
  "content":"...",
  "usage":{"prompt_tokens":...,"completion_tokens":...,"total_tokens":...}
}

If you omitted callback, GET /v1/jobs/<uuid> returns the same JSON after completion. That confirms you can handle long running llm tasks without blocking the initial request. To stress test, launch 50 concurrent curl calls; your API should stay at single-digit millisecond response times while the queue absorbs the load.

Operational notes

• Set worker concurrency based on GPU/rate limits, not blindly. RQ defaults to one thread; use multiple processes or a threaded worker class.
• Add dead-letter handling: if a job fails twice, move it to a failed queue and alert. Long prompts fail for transient reasons; don’t lose them silently.
• For per-task cost tracking, the gateway’s per-token usage metering (returned in usage) is enough to bill internally without extra instrumentation.
• Never let the API process execute the model. The moment you do, a traffic spike turns into a cascading timeout.
• Honor client routing directives if your gateway supports them; forward provider cache-control hints to cut repeat latency on similar prompts.

Following these steps gives you a clean separation: requests are accepted in milliseconds, inference runs in the background, and results land where the client needs them. That is the only sane way to handle long running llm tasks at production scale.