Parallel embedding requests: throughput scaling benchmarks

Parallel embedding request throughput scaling is the difference between a vector ingestion job that finishes in the time it takes to drink coffee and one that overruns your CI timeout. Most teams treat it as a simple “add more threads” problem, then hit invisible ceilings baked into model APIs and GPU schedulers. This analysis breaks down where the throughput gains actually come from and where they stop.

Why single-shot embedding wastes the pipe

Calling an embedding endpoint with one text per request is the default in quickstart code. It forces the provider to pay fixed overhead—TLS handshake, request parsing, scheduling—for every vector. Even on a fast network, that overhead dominates when payloads are small.

A single 512-token paragraph embedded alone might take 40 ms round trip. Embed the same paragraph inside a batch of 100 and per-item latency drops sharply because the fixed cost amortizes. The mistake is assuming that parallel single calls achieve the same amortization. They don’t; they multiply connection overhead.

Batch size and concurrency are separate levers

Provider batch limits

OpenAI’s embedding models accept up to 2048 inputs per request. That is a hard cap, not a suggestion. Local models served via TEI or vLLM often expose a similar max_batch_size config, typically 32–256 depending on memory.

If you send 1 item across 100 parallel connections, you are using 100x the connections to process what one batched request could handle at a fraction of the cost. Parallel embedding request throughput scaling only helps when you have more items than fit in a single batch, or when you are hitting token-rate limits that batching alone can’t bypass.

Client concurrency

Concurrency is how many in-flight requests your client maintains. It matters when:

• You exceed the per-request batch cap and must shard work.
• The provider rate-limits by requests per minute (RPM) or tokens per minute (TPM).
• Network round trips are the bottleneck, not compute.

But unbounded concurrency triggers connection pool exhaustion and TCP congestion. We have seen Python requests scripts spawn 500 threads and achieve lower total throughput than 32 workers because of lock contention and socket thrashing.

What an honest benchmark looks like

You cannot trust a throughput number without the exact payload shape. We standardized on:

• Fixed input: 256-token synthetic text chunks.
• Client: async Python using openai.AsyncOpenAI with httpx limits.
• Variable: concurrency level (1, 4, 16, 32, 64, 128) and batch size (1, 16, 64, 256).
• Metric: vectors per second averaged over 10k items.

No synthetic “lab conditions” with empty payloads. The goal was to mirror a real RAG ingestion worker.

from openai import AsyncOpenAI
import asyncio

client = AsyncOpenAI(base_url="https://api.openai.com/v1")
sem = asyncio.Semaphore(32)

async def embed_batch(texts):
    async with sem:
        resp = await client.embeddings.create(
            model="text-embedding-3-small",
            input=texts
        )
        return resp.data

async def worker(items, batch_size):
    tasks = []
    for i in range(0, len(items), batch_size):
        tasks.append(embed_batch(items[i:i+batch_size]))
    return await asyncio.gather(*tasks)

Connection pool sizing is not optional

An async semaphore limits your code’s concurrency, but the underlying HTTP client must allow the sockets. If you set sem to 32 but httpx defaults to 10 max connections, you serialize silently.

import httpx
from openai import AsyncOpenAI

http_client = httpx.AsyncClient(
    limits=httpx.Limits(max_connections=64, max_keepalive_connections=32)
)
client = AsyncOpenAI(http_client=http_client, base_url="https://api.openai.com/v1")

Match max_connections to your semaphore value plus headroom. For batch sizes above 64, keepalive connections reduce TLS rebuild cost.

The scaling curve is not linear

Parallel embedding request throughput scaling follows a classic saturation curve. At low concurrency (1–8), adding workers yields near-linear gains because you are filling idle provider capacity. Between 16 and 64, gains taper: you are now competing with other tenants or your own client’s connection limits. Beyond that, throughput flatlines or regresses.

The exact knee depends on the model and provider. For a shared cloud embedding endpoint, the TPM limit is usually the wall. For a dedicated GPU node running BGE-base, the wall is GPU compute and memory bandwidth—adding more client concurrency just increases queue depth server-side.

{
  "observation": "qualitative",
  "concurrency": [1, 8, 32, 128],
  "relative_throughput": ["1x", "6x", "9x", "9x"],
  "note": "Pattern illustrative of typical taper, not a measured benchmark"
}

The pattern holds across every embedding endpoint we have load-tested: diminishing returns, not collapse.

Batching beats parallelism for small workloads

If your total document count per job is under the provider batch cap, do not parallelize. Send one large request. It is simpler and uses fewer tokens per second of wall-clock.

# Bad: 1000 sequential single-item calls
for i in $(seq 1 1000); do curl -s api/embeddings -d '{"input":"doc'$i'"}'; done

# Good: one batched call with 1000 items (if under 2048)
curl -s api/embeddings -d '{"input":["doc1","doc2",...,"doc1000"]}'

When you exceed the cap, shard into batched requests and parallelize those shards. That hybrid is the only way parallel embedding request throughput scaling pays off.

Latency versus throughput tradeoff

Higher concurrency increases tail latency. A batch submitted with 128 other in-flight requests may wait in a provider queue. If your pipeline is offline ingestion, that is fine. If you are embedding at query time for a semantic cache, tail latency kills p99.

We recommend separate client profiles:

• Ingestion: high concurrency (32–64), large batches.
• Online: concurrency 1–4, small batches, aggressive timeouts.

Gateway and routing realities

An OpenAI-compatible gateway like n4n.ai can automatically fallback when a provider is rate-limited or degraded, but that does not exempt you from client-side tuning. The fallback adds a retry hop; if your semaphore is too large, you amplify load on the secondary provider and trigger its limits too.

Honor provider cache-control hints if the gateway forwards them—repeated identical embedding requests may hit a provider cache, making concurrency less critical for stable corpora.

Decisive takeaway

Tune batch size first, concurrency second. Measure your provider’s effective batch ceiling and TPM limit, then set an async semaphore to roughly 2–4x the number of batches needed to saturate that ceiling. Beyond that, you are adding latency without throughput. Parallel embedding request throughput scaling is real, but it is a finite resource bounded by physics and rate limits—treat it like a capacitor, not an infinite well.