Rate limits on embeddings endpoints turn simple batch jobs into fragile pipelines the moment you scale. This guide shows how to build a Node.js client that survives embeddings api rate limits node.js with structured retries, concurrency control, and provider fallback.
Step 1: Identify your rate limit dimensions
Embeddings APIs throttle on two independent axes: requests per minute (RPM) and tokens per minute (TPM). A single request carrying 200 short strings counts as one request against RPM but burns TPM proportional to the total input tokens, so a pipeline that looks safe on request count can still get cut off by token limits.
Most OpenAI-compatible services return the current state in response headers. Probe once at startup to read your ceilings:
import OpenAI from 'openai';
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const res = await client.embeddings.create({
model: 'text-embedding-3-small',
input: 'probe',
});
const headers = res.headers as Record<string, string>;
console.log('RPM limit:', headers['x-ratelimit-limit-requests']);
console.log('TPM limit:', headers['x-ratelimit-limit-tokens']);
console.log('Remaining requests:', headers['x-ratelimit-remaining-requests']);When you exceed a limit the server replies 429 Too Many Requests. The body is sparse; the useful signal is in headers:
{
"error": {
"type": "requests_limit_reached",
"message": "Rate limit reached for requests"
}
}Header retry-after gives seconds to wait; x-ratelimit-reset-requests gives seconds until the window resets. Client code must read these rather than hard-coding sleep values.
Step 2: Install a concurrency-limited queue
Firing Promise.all over ten thousand documents is the fastest way to hit a 429 wall. You need a leaky bucket. The p-queue package gives you interval-based throttling without writing your own timer logic.
npm install p-queue openaiCreate a queue sized to 80% of your observed RPM to leave headroom for other services sharing the key:
import PQueue from 'p-queue';
const RPM_LIMIT = 3000;
const safeCap = Math.floor(RPM_LIMIT * 0.8 / 60); // per second
export const embedQueue = new PQueue({
interval: 1000,
intervalCap: safeCap,
timeout: 30_000,
});If your bottleneck is TPM, estimate average tokens per item (e.g., 4 chars/token for English) and derive intervalCap from token budget instead. The queue also supports priority if some documents are user-facing and must jump the line.
Step 3: Implement exponential backoff with jitter
Retrying immediately on a 429 multiplies load exactly when the server is struggling. Wrap every call in a retry that honors retry-after and otherwise backs off exponentially with full jitter.
async function withRetry<T>(fn: () => Promise<T>, maxAttempts = 6): Promise<T> {
for (let attempt = 0; attempt <= maxAttempts; attempt++) {
try {
return await fn();
} catch (err: any) {
const status = err?.status;
if (status !== 429 && status !== 500 && status !== 503) throw err;
if (attempt === maxAttempts) throw err;
const retryAfter = err?.headers?.['retry-after'];
const waitMs = retryAfter
? parseInt(retryAfter, 10) * 1000
: Math.min(30_000, 400 * 2 ** attempt + Math.random() * 400);
await new Promise(r => setTimeout(r, waitMs));
}
}
throw new Error('unreachable');
}Embeddings requests are read-only and idempotent; retrying them is safe as long as you don’t double-count results downstream. Never retry on 401 or 400—those are permanent.
Step 4: Batch inputs to reduce request count
The embeddings endpoint accepts an array of strings in a single HTTP call. Batching shrinks RPM usage by the batch size but increases per-request TPM. For text-embedding-3-small, each input is capped at 8191 tokens, so chunk by both count and estimated token length.
function chunkByTokens(texts: string[], maxItems: number, maxTokens: number): string[][] {
const batches: string[][] = [];
let current: string[] = [];
let tokenEstimate = 0;
for (const t of texts) {
const est = Math.ceil(t.length / 4);
if (current.length >= maxItems || tokenEstimate + est > maxTokens) {
batches.push(current);
current = [];
tokenEstimate = 0;
}
current.push(t);
tokenEstimate += est;
}
if (current.length) batches.push(current);
return batches;
}
const batches = chunkByTokens(documents, 100, 300_000);Submit each batch to the queue:
const batchResults = await Promise.all(
batches.map(batch =>
embedQueue.add(() =>
withRetry(() =>
client.embeddings.create({
model: 'text-embedding-3-small',
input: batch,
})
)
)
)
);Step 5: Handle 429 and Retry-After headers explicitly
The retry wrapper covers transient throttling, but sustained limits need a circuit breaker. Track reset windows and pause the queue when the server signals a long recovery.
let circuitOpen = false;
embedQueue.on('error', (err: any) => {
const reset = parseInt(err?.headers?.['x-ratelimit-reset-requests'] ?? '0', 10);
if (reset > 30) {
circuitOpen = true;
embedQueue.pause();
setTimeout(() => {
circuitOpen = false;
embedQueue.start();
}, reset * 1000);
}
});While paused, new queue.add calls buffer in memory. If you have millions of items, persist the pending list to a Redis stream instead of growing the Node heap.
Step 6: Add fallback to a secondary provider
Single-provider rate limits are unavoidable during traffic spikes or provider incidents. Route to an OpenAI-compatible gateway that fronts multiple models. n4n.ai exposes one OpenAI-compatible endpoint for 240+ models and applies automatic fallback when a provider is rate-limited or degraded, which pairs well with your client-side queue.
Point a second client at the gateway and try it only after the primary exhausts retries:
const gatewayClient = new OpenAI({
apiKey: process.env.N4N_API_KEY,
baseURL: 'https://api.n4n.ai/v1',
});
async function embedWithFallback(batch: string[]) {
try {
return await withRetry(() =>
client.embeddings.create({ model: 'text-embedding-3-small', input: batch })
);
} catch (e: any) {
if (e?.status === 429 || e?.status === 503) {
return await withRetry(() =>
gatewayClient.embeddings.create({ model: 'text-embedding-3-small', input: batch })
);
}
throw e;
}
}Keep batching and queueing identical for the fallback path. The gateway may also forward provider cache-control hints, so repeated identical inputs can hit cache and avoid token metering entirely.
Step 7: Meter usage and verify the pipeline
Per-token metering is your only visibility into TPM headroom. Accumulate usage from each response and emit a structured log:
let totalTokens = 0;
let totalDocs = 0;
for (const r of batchResults) {
totalTokens += r.usage?.total_tokens ?? 0;
totalDocs += r.data.length;
}
console.log(JSON.stringify({ event: 'embed_complete', totalDocs, totalTokens }));Verify success with a deterministic test. Use nock to intercept the embeddings route and return a 429 twice, then 200:
import nock from 'nock';
nock('https://api.openai.com')
.post('/v1/embeddings')
.twice()
.reply(429, { error: { type: 'rate_limit' } }, { 'retry-after': '0' })
.post('/v1/embeddings')
.reply(200, { data: [{ embedding: [0.1] }], usage: { total_tokens: 5 } });Run the script on a 10-item sample. Assert:
- The process exits zero with all items embedded.
embedQueueconcurrency never exceededintervalCap(add a counter inactiveevents).- Fallback client received exactly one call when primary stayed at 429.
totalTokensequals the sum reported by mocked responses.
A green run proves your handling of embeddings api rate limits node.js is correct before you point it at production data.
Step 8: Tune for real traffic shapes
Batch jobs and online inference have opposite pressures. For a nightly backfill, maximize batch size and queue depth. For a live API that embeds user queries, keep batches small (size 8–16) and set intervalCap low to maintain tail latency.
If you consume a streaming source (Kafka, SQS), feed the queue continuously instead of loading all texts into memory:
for await (const msg of consumer) {
embedQueue.add(() => embedWithFallback([msg.body]));
}Monitor queue size with embedQueue.size and alert if it grows unbounded for five minutes—that indicates a limit you haven’t sized for.
Step 9: Keep embeddings api rate limits node.js boring
The goal is not cleverness; it’s a pipeline that logs a steady trickle of completions and never pages you at 2 a.m. because a provider tightened quotas. Queue, backoff, batch, fallback, meter. Wire those five pieces and the next rate limit event becomes a metric, not an incident.