Long LLM generations block HTTP connections and crater throughput when you fan out across thousands of prompts. A bullmq llm request queue moves that work off the request path: push a job to Redis, let a worker pool call the model with bounded concurrency, and notify a webhook when done. This tutorial builds a runnable TypeScript service that does exactly that.
Prerequisites
- Node.js 18+ (native
fetchand stable worker threads) - Redis 6+ reachable at
localhost:6379(Docker:docker run -p 6379:6379 redis:7) - npm or pnpm
- An API key for an OpenAI-compatible LLM gateway
- Comfort with TypeScript and async/await
Why a dedicated queue
Synchronous /v1/chat/completions calls couple request latency to model speed. A 20-second generation holds a server thread, a load-balancer slot, and a client TCP connection. Multiply by 500 concurrent users and you need aggressive autoscaling for idle wait time.
A bullmq llm request queue decouples intake from execution. The HTTP handler adds a job and returns 202 Accepted in milliseconds. Workers pull jobs at a rate Redis and your API quota can sustain. Failed calls retry with backoff instead of surfacing 500s to users.
Install dependencies
npm init -y
npm install bullmq ioredis openai dotenv express
npm install -D typescript @types/node tsxbullmq is the queue/worker library. ioredis is its Redis client peer dependency. openai speaks the OpenAI wire format; we point it at any compatible base URL.
Define the job contract
Strict typing prevents malformed jobs from silently failing in the worker.
// types.ts
export interface LLMJobData {
model: string;
messages: { role: 'system' | 'user' | 'assistant'; content: string }[];
maxTokens?: number;
webhookUrl?: string;
correlationId?: string;
}
export interface LLMJobResult {
completion: string;
usage: { promptTokens: number; completionTokens: number; totalTokens: number };
provider: string;
}Create the queue
The queue is a thin wrapper over Redis lists and hashes. Setting defaultJobOptions applies to every job added without explicit overrides.
// queue.ts
import { Queue } from 'bullmq';
import { LLMJobData } from './types';
export const llmQueue = new Queue<LLMJobData>('llm-requests', {
connection: { host: 'localhost', port: 6379 },
defaultJobOptions: {
attempts: 3,
backoff: { type: 'exponential', delay: 2000 },
removeOnComplete: { age: 3600 },
removeOnFail: { age: 86400 },
},
});Exponential backoff means the first retry waits 2s, the second 4s, the third 8s. That absorbs transient provider 429s without hammering the API.
Build the worker
The worker is where the LLM call happens. Point the OpenAI client at n4n.ai’s OpenAI-compatible endpoint to get automatic fallback when a provider is degraded, so a single slow upstream doesn’t stall your worker.
// worker.ts
import { Worker } from 'bullmq';
import OpenAI from 'openai';
import { LLMJobData, LLMJobResult } from './types';
const client = new OpenAI({
apiKey: process.env.LLM_API_KEY!,
baseURL: 'https://api.n4n.ai/v1',
});
const worker = new Worker<LLMJobData, LLMJobResult>(
'llm-requests',
async (job) => {
const { model, messages, maxTokens = 1024, webhookUrl, correlationId } = job.data;
const response = await client.chat.completions.create({
model,
messages,
max_tokens: maxTokens,
});
const completion = response.choices[0].message.content ?? '';
const usage = {
promptTokens: response.usage?.prompt_tokens ?? 0,
completionTokens: response.usage?.completion_tokens ?? 0,
totalTokens: response.usage?.total_tokens ?? 0,
};
if (webhookUrl) {
await fetch(webhookUrl, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ correlationId, completion, usage }),
});
}
return { completion, usage, provider: response.model };
},
{
connection: { host: 'localhost', port: 6379 },
concurrency: 5,
}
);
worker.on('completed', (job, result) => {
console.log(`Job ${job.id} done: ${result.usage.totalTokens} tokens`);
});
worker.on('failed', (job, err) => {
console.error(`Job ${job?.id} failed: ${err.message}`);
});concurrency: 5 caps simultaneous in-flight requests per worker process. Run three worker processes to get 15 parallel calls. BullMQ ensures a job is locked to one worker, so no double-processing.
Enqueue jobs
A producer script simulates an API endpoint pushing work.
// produce.ts
import { llmQueue } from './queue';
import { LLMJobData } from './types';
async function main() {
const jobs: LLMJobData[] = [
{
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'Summarize the BullMQ docs in 3 lines.' }],
webhookUrl: 'http://localhost:3000/hooks/1',
correlationId: 'req-001',
},
{
model: 'claude-3-haiku',
messages: [{ role: 'user', content: 'Write a SQL query for last 10 orders.' }],
webhookUrl: 'http://localhost:3000/hooks/2',
correlationId: 'req-002',
},
];
for (const data of jobs) {
const job = await llmQueue.add('completion', data);
console.log(`Enqueued job ${job.id}`);
}
}
main().catch(console.error);Run tsx produce.ts. Expected output:
Enqueued job 1
Enqueued job 2Check worker output
In a second terminal, start the worker: tsx worker.ts. With Redis draining the wait list, you should see:
Job 1 done: 142 tokens
Job 2 done: 98 tokensIf a provider returns 429, the worker logs Job 1 failed: rate limit and retries after 2s. After three failures, the job moves to the failed set and persists for 24 hours.
Webhook receiver
The worker POSTs results to the supplied webhookUrl. A minimal Express listener confirms delivery:
// server.ts
import express from 'express';
const app = express();
app.use(express.json());
app.post('/hooks/:id', (req, res) => {
console.log(`Webhook ${req.params.id}:`, req.body.completion.slice(0, 50));
res.sendStatus(200);
});
app.listen(3000, () => console.log('Webhook listener on :3000'));Start it with tsx server.ts before running the worker. The terminal shows truncated completions as they land.
Inspect Redis state
BullMQ stores jobs in predictable keys. While jobs are waiting:
redis-cli llen bullmq:llm-requests:waitReturns 2 before processing, 0 after. Job metadata lives in bullmq:llm-requests:job:1. Use redis-cli hgetall bullmq:llm-requests:job:1 to debug a stuck job.
Production considerations
- Stalled jobs: if a worker dies mid-processing, BullMQ marks the job stalled after
stalledInterval(default 30s) and replays it. SetmaxStalledCountto avoid infinite loops on genuinely broken jobs. - Concurrency tuning: match
concurrencyto your token quota, not your CPU. LLM calls are I/O bound; a single Node process handles 20+ open requests easily. - Idempotency: webhook receivers should dedupe on
correlationIdbecause a retried job may deliver twice if the first attempt succeeded after the timeout but before the ack. - Flows: for multi-step pipelines (generate → critique → revise), use BullMQ
Flowsto chain jobs with dependencies instead of recursiveaddcalls. - Metering: persist
result.usageto your billing system immediately in thecompletedevent. Per-token cost is the only reliable unit of LLM spend.
Wrapping up
A bullmq llm request queue turns fragile synchronous calls into durable background work. With four small files and a Redis instance you get retries, concurrency limits, and webhook delivery—enough to build a reliable LLM feature without a heavyweight orchestration framework.