How to rate-limit requests with the OpenAI Node.js SDK

Hitting OpenAI’s 429 Too Many Requests response is a rite of passage. If you’re shipping a service that calls the API, you need to openai node.js sdk rate limit requests on the client before the provider does it for you. This post walks through a concrete, runnable pattern: token-bucket throttling, bounded concurrency, and retry-with-backoff wrapped around the official SDK.

Step 1: Install and initialize the OpenAI Node.js SDK

Start with the official package. As of v4, the import is named OpenAI and the client is constructed with an API key from the environment.

npm install openai

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

// Quick smoke test
const res = await client.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'ping' }],
});
console.log(res.choices[0].message.content);

The SDK itself does zero throttling. It will fire whatever you call, whenever you call it. The openai node.js sdk rate limit requests responsibility falls entirely on your code.

Step 2: Inspect the rate limit headers

OpenAI sends informative headers on every response, and a retry-after header on 429s. Knowing these lets you build adaptive logic, but even a static config needs to respect them.

try {
  await client.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: 'headers test' }],
  });
} catch (err) {
  const h = err.response?.headers;
  console.log({
    limit: h?.['x-ratelimit-limit-requests'],
    remaining: h?.['x-ratelimit-remaining-requests'],
    reset: h?.['x-ratelimit-reset-requests'],
    retryAfter: h?.['retry-after'],
  });
}

Common tiers expose per-minute request and token quotas. Pick a conservative static rate (e.g., 60 requests/minute = 1 req/sec) for your bucket until you verify with live traffic.

Step 3: Implement a token-bucket limiter

A token bucket is the right primitive: it allows bursts up to capacity and refills at a steady rate. Below is a minimal async implementation with no dependencies.

class TokenBucket {
  private tokens: number;
  private lastRefill: number;

  constructor(
    private capacity: number,
    private refillPerSec: number,
  ) {
    this.tokens = capacity;
    this.lastRefill = Date.now();
  }

  async take(): Promise<void> {
    while (true) {
      this.refill();
      if (this.tokens >= 1) {
        this.tokens -= 1;
        return;
      }
      await new Promise((r) => setTimeout(r, 50));
    }
  }

  private refill(): void {
    const now = Date.now();
    const elapsedSec = (now - this.lastRefill) / 1000;
    this.tokens = Math.min(
      this.capacity,
      this.tokens + elapsedSec * this.refillPerSec,
    );
    this.lastRefill = now;
  }
}

// 1 request/sec, burst of 5
const bucket = new TokenBucket(5, 1);

This class is the core of openai node.js sdk rate limit requests. Every outbound call must await bucket.take() first.

Step 4: Wrap SDK calls with limiter and retry

The bucket prevents you from sending too fast. But races, clock skew, or misconfigured tiers still produce 429s. Wrap the call with a bounded retry that honors retry-after.

async function chatWithLimit(
  client: OpenAI,
  bucket: TokenBucket,
  body: OpenAI.Chat.ChatCompletionCreateParamsNonStreaming,
  maxRetries = 3,
) {
  await bucket.take();
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await client.chat.completions.create(body);
    } catch (err: any) {
      if (err?.status === 429) {
        const ra = err?.response?.headers?.['retry-after'];
        const backoff = ra ? Number(ra) : 2 ** attempt;
        await new Promise((r) => setTimeout(r, backoff * 1000));
        continue;
      }
      throw err;
    }
  }
  throw new Error('Exhausted retries on 429');
}

Note the SDK exposes err.status and err.response.headers. Do not assume JSON body shape; the headers are authoritative.

Step 5: Bound concurrency with a queue

A token bucket serializes rate but not parallelism. If you Promise.all 100 calls, they all wait in the bucket but then execute concurrently the moment tokens free up, which can still trip concurrency limits. Use p-queue to cap in-flight requests.

npm install p-queue

import PQueue from 'p-queue';

const queue = new PQueue({ concurrency: 5 });

function chatThrottled(
  body: OpenAI.Chat.ChatCompletionCreateParamsNonStreaming,
) {
  return queue.add(() => chatWithLimit(client, bucket, body));
}

Now you have two independent controls: bucket enforces requests-per-second; queue enforces max parallel connections. Together they fully cover openai node.js sdk rate limit requests for a single process.

Step 6: Verify your rate limiting works

Write a script that hammers the API faster than the limit and confirm no unhandled 429 escapes.

const start = Date.now();
const tasks = Array.from({ length: 20 }, (_, i) =>
  chatThrottled({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: `ping ${i}` }],
  }),
);

const results = await Promise.allSettled(tasks);
const failed = results.filter((r) => r.status === 'rejected');
console.log(`Completed in ${Date.now() - start}ms`);
console.log(`Failed requests: ${failed.length}`);

if (failed.length > 0) {
  console.error('Unexpected failures', failed.slice(0, 3));
  process.exit(1);
}

Success criteria:

• The script finishes with zero rejected promises.
• Wall-clock time is roughly 20 requests / 1 req-per-sec ≈ 20s (plus queue concurrency speedup only if tokens allow).
• If you log inside bucket.take(), you see sleeps when burst capacity is exhausted.

If you still see 429s after retries, your refillPerSec is above the actual provisioned rate. Lower it by 20% and re-test.

Production considerations

Distributed workers

The in-memory TokenBucket is per-process. Two Node instances behind a load balancer will each think they have full quota and collectively exceed it. For multi-process or multi-host deployments, move the bucket to Redis (or use a centralized limiter like rate-limiter-flexible).

Gateway offload

Client-side throttling is necessary but not sufficient when providers degrade. If you front your calls with an OpenAI-compatible endpoint such as n4n.ai, you get automatic fallback when a provider is rate-limited or degraded, plus per-token usage metering and honored cache-control hints. That complements the client pattern above: you still openai node.js sdk rate limit requests locally to avoid wasteful retries, while the gateway absorbs regional provider outages.

Observability

Export metrics for bucket.wait_ms, queue.size, and 429_retry_count. A sudden rise in bucket wait time means your assumed limit is lower than reality. Treat rate limits as a moving target and alert on retry exhaustion.