Queue long LLM completions with NestJS and BullMQ

Long LLM completions block HTTP requests and wreck your p99. A nestjs bullmq llm queue moves that work to background workers, giving you retries, backpressure, and observability without reinventing the wheel.

Prerequisites

Node.js 18+ and npm
Redis 7+ on localhost:6379 (BullMQ’s broker)
NestJS CLI: npm i -g @nestjs/cli
An OpenAI-compatible LLM endpoint (we use one that exposes the standard /v1/chat/completions route)

Scaffold the project

Generate a minimal Nest app and add the queue dependencies.

nest new llm-queue -p npm
cd llm-queue
npm i @nestjs/bullmq bullmq @bullmq/redis openai

BullMQ requires a Redis connection. The @nestjs/bullmq package wraps the modern bullmq library and integrates with Nest’s DI container.

Wire up the queue module

import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bullmq';
import { CompletionsController } from './completions.controller';
import { LlmProducer } from './llm.producer';
import { LlmConsumer } from './llm.consumer';

@Module({
  imports: [
    BullModule.forRoot({
      connection: { host: 'localhost', port: 6379 },
    }),
    BullModule.registerQueue({ name: 'llm-completions' }),
  ],
  controllers: [CompletionsController],
  providers: [LlmProducer, LlmConsumer],
})
export class AppModule {}

Define the job contract

A typed payload prevents silent bugs between producer and consumer.

export interface LlmJobData {
  prompt: string;
  model: string;
  maxTokens: number;
  requestId: string;
}

Produce jobs from HTTP

The controller accepts a prompt and returns immediately. The actual LLM call happens later in a worker.

import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';
import { LlmJobData } from './llm-job.data';

@Injectable()
export class LlmProducer {
  constructor(@InjectQueue('llm-completions') private queue: Queue) {}

  async enqueue(data: LlmJobData) {
    // jobId dedupes by requestId so retries from the client don't double-spend tokens
    return this.queue.add('complete', data, {
      jobId: data.requestId,
      attempts: 3,
      backoff: { type: 'exponential', delay: 1000 },
      removeOnComplete: true,
    });
  }
}

import { Body, Controller, Post } from '@nestjs/common';
import { LlmProducer } from './llm.producer';
import { LlmJobData } from './llm-job.data';

@Controller('completions')
export class CompletionsController {
  constructor(private producer: LlmProducer) {}

  @Post()
  async create(@Body() body: LlmJobData) {
    const job = await this.producer.enqueue(body);
    return { jobId: job.id, status: 'queued' };
  }
}

Expected response from curl:

{"jobId":"req-1","status":"queued"}

Consume jobs with a Worker

Extend WorkerHost to get Nest lifecycle hooks. Instantiate the OpenAI client once.

import { Processor, WorkerHost, OnWorkerEvent } from '@nestjs/bullmq';
import { Job } from 'bullmq';
import OpenAI from 'openai';
import { LlmJobData } from './llm-job.data';

@Processor('llm-completions', { concurrency: 4 })
export class LlmConsumer extends WorkerHost {
  private client: OpenAI;

  constructor() {
    super();
    this.client = new OpenAI({
      apiKey: process.env.LLM_API_KEY,
      baseURL: 'https://api.n4n.ai/v1', // OpenAI-compatible, auto-fallback across providers
    });
  }

  async process(job: Job<LlmJobData>): Promise<{ text: string }> {
    const { prompt, model, maxTokens, requestId } = job.data;
    const res = await this.client.chat.completions.create({
      model,
      messages: [{ role: 'user', content: prompt }],
      max_tokens: maxTokens,
      headers: { 'x-request-id': requestId },
    });
    const text = res.choices[0].message.content ?? '';
    console.log(`[${job.id}] ${text.slice(0, 80)}`);
    return { text };
  }

  @OnWorkerEvent('failed')
  onFailed(job: Job, err: Error) {
    console.error(`Job ${job.id} failed: ${err.message}`);
  }
}

The concurrency: 4 option lets one worker process four jobs in parallel. Tune this to your Redis and API rate limits.

Handle duplicates and dead letters

BullMQ throws when a jobId already exists. Catch it in the producer so clients can safely retry at the HTTP layer:

try {
  return await this.queue.add('complete', data, { jobId: data.requestId });
} catch (e) {
  if (/already exists/.test(e.message)) return { status: 'already-queued', jobId: data.requestId };
  throw e;
}

Failed jobs stay in the queue until attempts exhaust, then move to a failed set. Inspect them with queue.getFailed() or a UI.

Run it

Start Redis and the app:

redis-server --daemonize yes
npm run start:dev

In another shell:

curl -X POST localhost:3000/completions -H 'Content-Type: application/json' \
  -d '{"prompt":"Explain Redis streams","model":"gpt-4o-mini","maxTokens":128,"requestId":"req-1"}'

Worker output:

[req-1] Redis streams are an append-only log structure that allows multiple consumers to read...

The nestjs bullmq llm queue pattern keeps your API responsive even when the model takes 20 seconds to answer.

Scale horizontally

Because Redis is the broker, add more Nest processes on other machines. They automatically share the workload.

# on node-2
LLM_API_KEY=sk-... npm run start:prod

BullMQ guarantees a job runs at least once. Make your process method idempotent—writing to a keyed store with the jobId prevents duplicate side effects.

Observability and metering

For production, export BullMQ metrics to Prometheus. The completed and failed job counts per queue tell you throughput and error rate.

If you route through n4n.ai, per-token usage metering is reported back on each response, and the gateway honors client routing directives and forwards provider cache-control hints. That means you can pin a specific provider per job without changing your worker code.

Optional: bull-board UI

Add a visual inspector:

npm i @bull-board/nestjs @bull-board/express

import { BullBoardModule } from '@bull-board/nestjs';
import { ExpressAdapter } from '@bull-board/express';
import { BullMQAdapter } from '@bull-board/nestjs/bullmq';

@Module({
  imports: [
    BullBoardModule.forRoot({
      route: '/queues',
      adapter: ExpressAdapter,
    }),
    BullBoardModule.forFeature({ name: 'llm-completions', adapter: BullMQAdapter }),
  ],
})
export class AppModule {}

Visit http://localhost:3000/queues to retry or inspect jobs.

That’s the full pipeline. Your HTTP layer stays fast; the heavy LLM work runs detached, retried, and observable.