Async job patterns for LLM APIs in Node.js

Long-running LLM inferences break naive request/response cycles. This tutorial builds async job patterns nodejs llm that survive provider latency, rate limits, and partial failures using a queue, worker, and webhook callback. You will stand up a minimal but production-shaped system in under 200 lines of Node.js.

Prerequisites

• Node.js 18+ (native fetch, ESM support)
• A running Redis instance (Docker: docker run -p 6379:6379 redis)
• An OpenAI-compatible API key. We’ll point the SDK at https://api.n4n.ai/v1 — one endpoint that fronts 240+ models with automatic fallback when a provider is degraded.
• Install dependencies:

npm init -y
npm install bullmq openai express dotenv

Create a .env file:

LLM_API_KEY=sk-your-key

Architecture

The flow is simple:

1. Client POSTs a prompt + callbackUrl to /jobs.
2. API enqueues a job in Redis via BullMQ and returns 202 with a jobId.
3. A separate worker pulls the job, calls the LLM, then POSTs the result to callbackUrl.
4. If the worker crashes or the provider rate-limits, BullMQ retries with backoff.

These async job patterns nodejs llm decouple request latency from inference time and let you scale workers independently of your HTTP tier.

Step 1: Queue and worker

Create jobQueue.mjs. This module exports the queue and spins up a worker.

import { Queue, Worker } from 'bullmq';
import { OpenAI } from 'openai';
import dotenv from 'dotenv';
dotenv.config();

const connection = { host: 'localhost', port: 6379 };

export const llmQueue = new Queue('llm-jobs', { connection });

const openai = new OpenAI({
  apiKey: process.env.LLM_API_KEY,
  baseURL: 'https://api.n4n.ai/v1', // OpenAI-compatible, 240+ models, auto fallback
});

const handler = async (job) => {
  const { prompt, model, callbackUrl } = job.data;
  const completion = await openai.chat.completions.create({
    model,
    messages: [{ role: 'user', content: prompt }],
  });
  const result = completion.choices[0].message.content;
  await fetch(callbackUrl, {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ jobId: job.id, result }),
  });
  return { ok: true };
};

new Worker('llm-jobs', handler, {
  connection,
  attempts: 3,
  backoff: { type: 'exponential', delay: 1000 },
});

console.log('Worker started');

Run node jobQueue.mjs. Expected output:

Worker started

The worker now blocks on Redis, ready to process jobs.

Step 2: HTTP ingestion API

Create server.mjs to accept jobs and expose a polling fallback.

import express from 'express';
import { llmQueue } from './jobQueue.mjs';

const app = express();
app.use(express.json());

app.post('/jobs', async (req, res) => {
  const { prompt, model = 'gpt-4o-mini', callbackUrl } = req.body;
  if (!prompt || !callbackUrl) {
    return res.status(400).json({ error: 'prompt and callbackUrl required' });
  }
  const job = await llmQueue.add('completion', { prompt, model, callbackUrl });
  res.status(202).json({ jobId: job.id });
});

app.get('/jobs/:id', async (req, res) => {
  const job = await llmQueue.getJob(req.params.id);
  if (!job) return res.status(404).json({ error: 'not found' });
  const state = await job.getState();
  res.json({ state, result: job.returnvalue });
});

app.listen(3000, () => console.log('API on :3000'));

In a separate terminal, start the API:

node server.mjs

Test with curl:

curl -X POST localhost:3000/jobs \
  -H 'content-type: application/json' \
  -d '{"prompt":"Explain Raft in one sentence","callbackUrl":"http://localhost:3000/webhook","model":"claude-3-haiku"}'

Expected response:

{"jobId":"1"}

Step 3: Webhook receiver and idempotency

Add a webhook route to server.mjs (or a separate service). Clients often cannot open ports, so your server pushes to them; here we loop back for demo.

const seen = new Set();

app.post('/webhook', (req, res) => {
  const { jobId, result } = req.body;
  if (seen.has(jobId)) return res.sendStatus(200); // idempotent
  seen.add(jobId);
  console.log(`Job ${jobId} completed: ${result.slice(0, 60)}...`);
  res.sendStatus(200);
});

When the worker finishes, your console shows:

Job 1 completed: Raft is a consensus algorithm that keeps a distributed log...

If the client’s callbackUrl is down, BullMQ will not retry the webhook (the job succeeded). For strict delivery, move the fetch into a separate retryable job or use a transactional outbox. The async job patterns nodejs llm shown here keep the LLM call retryable; webhook delivery is best-effort unless you add a dead-letter queue.

Step 4: Polling as a webhook alternative

Not every environment can receive HTTP pushes. The /jobs/:id endpoint above already supports polling. Example:

curl localhost:3000/jobs/1

While pending:

{"state":"waiting","result":null}

After completion:

{"state":"completed","result":{"ok":true}}

Pollers should cache the final state and stop. Combine polling with webhooks: push when possible, poll as fallback.

Step 5: Passing provider hints

OpenAI-compatible gateways forward cache-control and routing directives. If you want to force a specific provider or enable prompt caching, pass headers via the SDK:

await openai.chat.completions.create(
  {
    model,
    messages: [{ role: 'user', content: prompt }],
  },
  {
    headers: {
      'x-n4n-routing': 'anthropic',
      'x-cache-control': 'ephemeral',
    },
  }
);

The worker ignores unknown headers gracefully if the endpoint doesn’t support them. In our setup the gateway honors client routing directives and forwards provider cache-control hints, so the same code works across models without branching.

Step 6: Production hardening

The code so far is a skeleton. Before shipping, address these:

• Redis durability: use AOF persistence or a managed Redis. BullMQ jobs vanish on flush.
• Worker isolation: run jobQueue.mjs as its own process (Docker container, systemd unit). Never run workers inside the HTTP server in production — a memory leak in inference handling kills your ingress.
• Signed webhooks: verify an HMAC signature on callbackUrl receipt. Otherwise spoofed POSTs mark jobs done.
• Concurrency: set concurrency on the Worker based on provider rate limits. Default is 1; bump to 10 if tokens allow.
• Job TTL: llmQueue.add(..., { removeOnComplete: 3600 }) to avoid Redis bloat.

A hardened worker constructor:

new Worker('llm-jobs', handler, {
  connection,
  attempts: 5,
  backoff: { type: 'exponential', delay: 2000 },
  concurrency: 8,
  removeOnComplete: 3600,
  removeOnFail: 24 * 3600,
});

Step 7: Observing token usage

The completion response includes usage. Meter it inside the handler:

const { prompt_tokens, completion_tokens } = completion.usage ?? {};
console.log(`job ${job.id}: ${prompt_tokens}+${completion_tokens} tokens`);

If your gateway provides per-token usage metering, you can skip local accounting and reconcile from its dashboard. Either way, emit these metrics to Prometheus or your APM so you catch slow providers early.

Recap

You now have a runnable skeleton for async job patterns nodejs llm:

• HTTP API enqueues jobs and returns immediately.
• BullMQ worker calls an OpenAI-compatible endpoint with retries.
• Results push to a webhook; clients can also poll.
• Routing and cache hints pass through unchanged.

Copy the files, point LLM_API_KEY at your gateway, and you have a resilient inference backend that won’t time out under load.