Rate limit LLM calls in NestJS with @nestjs/throttler

LLM endpoints are expensive and strictly metered by providers; a single runaway client can burn your budget or trip upstream 429s. Applying nestjs throttler llm rate limiting at the API gateway layer lets you cap request rates per user, per route, and per model before the call ever leaves your infrastructure.

Step 1: Install dependencies and scaffold the module

Start with a standard NestJS service. If you don’t have one, generate it with the CLI:

npm i -g @nestjs/cli
nest new llm-gateway
cd llm-gateway
npm i @nestjs/throttler @nestjs/config

@nestjs/throttler ships a guard, module, and decorator set that integrate with Nest’s dependency injection. For LLM workloads you’ll usually want Redis-backed storage so limits survive restarts and apply across instances. Add it now:

npm i @nestjs/throttler-storage-redis ioredis

The base nestjs throttler llm rate limiting setup is identical to any other route protection, but the thresholds and error handling differ because LLM calls are slow and costly.

Step 2: Register ThrottlerModule with sensible defaults

Wire the module in app.module.ts. Use forRootAsync to pull TTL and limit from env, and point storage at Redis:

import { ThrottlerModule } from '@nestjs/throttler';
import { ThrottlerStorageRedisService } from '@nestjs/throttler-storage-redis';
import Redis from 'ioredis';

@Module({
  imports: [
    ThrottlerModule.forRootAsync({
      useFactory: () => {
        const redis = new Redis(process.env.REDIS_URL);
        return {
          throttlers: [{ ttl: 60000, limit: 20 }],
          storage: new ThrottlerStorageRedisService(redis),
        };
      },
    }),
  ],
})
export class AppModule {}

This caps every route to 20 requests per minute globally. For LLM routes you’ll override per endpoint because a chat completion is not equivalent to a health check.

Step 3: Create an LLM-aware throttler guard

The default ThrottlerGuard keys on IP or user id. For LLM calls, key on the authenticated subject and the target model. Subclass the guard:

import { ThrottlerGuard } from '@nestjs/throttler';
import { ExecutionContext, Injectable } from '@nestjs/common';

@Injectable()
export class LLMThrottlerGuard extends ThrottlerGuard {
  protected async getTracker(req: any): Promise<string> {
    const userId = req.user?.id ?? req.ip;
    const model = req.body?.model ?? 'default';
    return `${userId}:${model}`;
  }
}

Bind it globally or per controller. Global binding in main.ts:

app.useGlobalGuards(new LLMThrottlerGuard());

Now nestjs throttler llm rate limiting enforces isolation between models: a user hammering gpt-4o won’t exhaust their quota for gpt-4o-mini.

Step 4: Build the LLM proxy controller

Create a controller that forwards to an OpenAI-compatible endpoint. If you proxy through n4n.ai, its automatic fallback covers upstream provider degradation, but nestjs throttler llm rate limiting still caps your client traffic.

import { Controller, Post, Body } from '@nestjs/common';
import { Throttle } from '@nestjs/throttler';

@Controller('llm')
export class LLMController {
  @Post('chat')
  @Throttle({ default: { ttl: 60000, limit: 5 } })
  async chat(@Body() body: { model: string; prompt: string }) {
    const res = await fetch('https://api.n4n.ai/v1/chat/completions', {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        authorization: `Bearer ${process.env.LLM_API_KEY}`,
      },
      body: JSON.stringify({
        model: body.model,
        messages: [{ role: 'user', content: body.prompt }],
      }),
    });
    return res.json();
  }
}

fetch is fine in Node 18+. The @Throttle decorator overrides the global 20/min with a stricter 5/min per model per user thanks to the guard in Step 3.

Step 5: Apply differentiated limits per route type

Embeddings and moderation calls are cheaper than generation. Reflect that with separate decorators:

@Post('embed')
@Throttle({ default: { ttl: 60000, limit: 50 } })
async embed(@Body() body: { input: string }) {
  // call embedding endpoint
}

@Post('chat')
@Throttle({ default: { ttl: 60000, limit: 5 } })
async chat(...) { ... }

This granular nestjs throttler llm rate limiting prevents a bulk embedding job from starving interactive chat users.

Step 6: Return proper 429 semantics

@nestjs/throttler throws ThrottlerException (HTTP 429) when limits exceed. Customize the response to include Retry-After and a JSON body:

import { ExceptionFilter, Catch, ArgumentsHost } from '@nestjs/common';
import { ThrottlerException } from '@nestjs/throttler';

@Catch(ThrottlerException)
export class ThrottlerFilter implements ExceptionFilter {
  catch(exception: ThrottlerException, host: ArgumentsHost) {
    const res = host.switchToHttp().getResponse();
    res.status(429).setHeader('Retry-After', '60');
    res.json({ error: 'rate_limited', message: exception.message });
  }
}

Register the filter in main.ts with app.useGlobalFilters(new ThrottlerFilter()). Clients then back off correctly instead of retrying immediately.

Step 7: Verify the limit with a loop

Write a quick bash loop to fire 10 requests at the chat route and observe the 429:

for i in {1..10}; do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:3000/llm/chat \
    -H 'content-type: application/json' \
    -d '{"model":"gpt-4o-mini","prompt":"hi"}'
done

The first five return 200; the sixth returns 429. Check the header:

curl -i -X POST localhost:3000/llm/chat -d '{"model":"gpt-4o-mini","prompt":"hi"}' | grep retry-after

You should see Retry-After: 60. That confirms nestjs throttler llm rate limiting is active.

Production notes: tokens, not just requests

Throttler counts HTTP requests. A 10k-token prompt costs far more than a 10-token one. For tighter cost control, extend the guard to estimate tokens from req.body and weight the limit:

protected async getTracker(req: any): Promise<string> {
  const estTokens = Math.ceil((req.body?.prompt?.length ?? 0) / 4);
  req.throttleWeight = estTokens;
  return `${req.user?.id}:${req.body?.model}`;
}

Then override getLimit or use a custom storage that decrements by weight. Redis storage supports this if you track counters manually.

Run at least two instances behind a load balancer to confirm Redis storage shares state. Without it, each node enforces its own limit and the effective ceiling scales with node count.

That’s the full path: install, configure, subclass, proxy, decorate, filter, verify. Your LLM routes are now bounded by client, model, and route—no more surprise provider bills.