Batch size and its effect on embedding throughput

Batch size embedding throughput is the single most leveraged knob for squeezing cost out of vectorization pipelines, yet most teams leave it at one. The thesis here is simple: stacking inputs into batches of 32–128 typically yields order-of-magnitude throughput gains over single-item calls, but the curve flattens hard past the point where GPU memory bandwidth saturates, and some providers enforce hidden caps that silently truncate your wins.

Why batch size embedding throughput matters

Embedding generation is embarrassingly parallel across the batch dimension. A transformer encoder applies the same weight matrices to every token in a batch; the matrix multiplies are the dominant cost. Launching a CUDA kernel for one sequence vs sixty-four sequences does not cost 64x the compute—it costs marginally more due to larger matrix dimensions, but the fixed overhead of kernel launch, memory allocation, and API round-trip is amortized.

If you call an embedding endpoint once per document, you pay that fixed overhead per item. At 10 million documents, that overhead is the difference between a job that finishes in minutes and one that runs for hours, or between a trivial cloud bill and a painful one.

The second-order effect is network efficiency. A single HTTP request carrying one sentence still bears TLS handshake, headers, and JSON parsing overhead. Batching 100 sentences into one request spreads that tax across 100 items.

The hardware reality: compute vs memory bandwidth

Most embedding models are small relative to LLMs—often 100M–1B parameters. On modern GPUs, they are memory-bandwidth bound, not compute bound. The weights must be streamed from HBM to the compute units for every forward pass. Batching helps because larger batches increase the arithmetic intensity: more useful math per byte loaded.

Consider a layer with weight matrix W of size [hidden, hidden]. For a single sequence of length L, the matmul is [L, hidden] x [hidden, hidden] → [L, hidden]. For batch B, it becomes [B*L, hidden] x [hidden, hidden]. The weight bytes loaded are identical; the FLOPs scale with B. Until you exceed the GPU’s ability to keep the SMs fed, throughput scales near linearly with B.

Past saturation, you hit two walls:

1. Activation memory: intermediates of size B * L * hidden * layers * 2 bytes (fp16) must reside in GPU RAM.
2. Kernel efficiency limits: some kernels peak at specific tile sizes; awkward batch shapes waste cycles.

Sequence length shifts the knee

The attention block is O(B * L^2). For a 384-dim model with L=32, attention is cheap. At L=512, doubling batch size quadruples attention cost. In practice, long-document embedding peaks at smaller batches—often 16–32—while 32-token queries can scale to 256+ before flattening.

Measuring batch size embedding throughput correctly

Naive microbenchmarks lie. You must measure end-to-end: client serialization, network, server queue, inference, response parsing. A local time.time() around a loop is insufficient if you ignore network variance and provider queueing.

A minimal benchmarking harness

Below is a reproducible pattern using the OpenAI Python client. It works against any OpenAI-compatible server, including a gateway that fronts multiple providers.

import time, random
from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

def make_texts(n, length=32):
    return [" ".join(["word"]*length) for _ in range(n)]

def bench(model, batch_size, total=2000):
    texts = make_texts(total)
    start = time.time()
    for i in range(0, total, batch_size):
        client.embeddings.create(
            model=model,
            input=texts[i:i+batch_size]
        )
    elapsed = time.time() - start
    return total / elapsed  # embeddings per second

for bs in [1, 8, 32, 64, 128, 256]:
    rate = bench("text-embedding-3-small", bs)
    print(f"batch={bs:4d}  {rate:7.1f} emb/s")

When experimenting across model providers, an OpenAI-compatible endpoint that addresses 240+ models lets you swap the model string without rewriting client code, which simplifies comparing batch size embedding throughput across architectures.

For on-prem hardware, sentence-transformers exposes batch_size directly:

from sentence_transformers import SentenceTransformer
import time

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = ["sample clause"] * 5000

for bs in [1, 16, 64, 256, 512]:
    t0 = time.time()
    for i in range(0, len(texts), bs):
        model.encode(texts[i:i+bs], batch_size=bs, show_progress_bar=False)
    print(bs, len(texts)/(time.time()-t0), "emb/s")

Run each configuration three times and discard the first as warm-up. Plot emb/s against batch size; the curve tells you where your hardware or provider caps out.

Diminishing returns and where the cliff is

On a T4 GPU with all-MiniLM-L6-v2 (384-dim, 6 layers), throughput often climbs steeply from bs=1 to bs=32, then plateaus by bs=128. Beyond bs=256, you may see regression as the encoder’s attention step starts to dominate for longer sequences.

For hosted APIs, the cliff is frequently earlier because the provider multiplexes your batch across shared hardware. A batch of 512 might be split server-side into 64-item chunks, negating your client-side optimization. You can detect this by plotting emb/s vs bs: if the slope dies at a suspiciously round number, suspect server-side chunking.

Tradeoffs: latency, memory, and provider limits

Throughput and latency are opposite ends of the same dial. A batch of 256 sentences might give 15x the embeddings per second of batch=1, but the first embedding in that batch waits for the slowest item to finish. If your service embeds user queries in-line, a 256-batch forces 200 ms p99 latency instead of 20 ms.

Client-side batching vs server-side limits

Hosted embedding endpoints impose constraints:

• Max items per request (commonly 2048 for OpenAI-compatible APIs, but smaller for some open models).
• Max total tokens per request (e.g., 300k).
• Rate limits in tokens/min.

If you blindly queue 1000 docs into one request and hit a 100-doc cap, you get a 400. Wrap batching in a clamp:

def chunked(batch, max_items=100, max_tokens=250_000):
    cur, tok = [], 0
    for item in batch:
        t = len(item.split())
        if len(cur) >= max_items or tok + t > max_tokens:
            yield cur
            cur, tok = [], 0
        cur.append(item); tok += t
    if cur: yield cur

Memory on the client also matters: holding 10k texts in RAM before batching is fine; holding 10M may require streaming from disk.

Handling partial failures in large batches

When a batch of 200 fails at item 150 due to a provider 429 or a malformed input, you do not want to redo all 200. Some SDKs return partial data; others throw. Design idempotent retries that re-submit only the missing indices.

def embed_with_retry(client, model, items, max_items=100):
    out = [None] * len(items)
    queue = list(range(len(items)))
    while queue:
        batch_idx = queue[:max_items]
        try:
            resp = client.embeddings.create(
                model=model,
                input=[items[i] for i in batch_idx]
            )
            for j, d in zip(batch_idx, resp.data):
                out[j] = d.embedding
            queue = queue[max_items:]
        except Exception:
            # shrink batch on error to isolate bad items
            if max_items == 1:
                queue.pop(0)
            else:
                max_items //= 2
    return out

This pattern keeps batch size embedding throughput high while preventing a single bad record from poisoning a 10k-item job.

Practical batch sizing recommendations

• Offline indexing: Use the largest batch that fits in provider limits and keeps GPU memory under 90%. Start at 128, sweep to 512.
• Near-real-time (seconds): Batch over a short time window (e.g., 100 ms) and flush. This trades a little latency for 5–10x throughput.
• Synchronous user requests: Batch size 1. Do not micro-batch across users unless you control the latency SLA.

For sequence length, longer texts shift the optimal batch down. If your average doc is 512 tokens, bs=64 may saturate where bs=256 worked for 32-token tweets. Always re-measure when you change model dimensions or host.

Decisive takeaway

Batch size embedding throughput follows a predictable arc: linear gains from amortized overhead, then a knee where memory bandwidth saturates, then a flat or declining tail due to attention cost and provider chunking. Set batch size to the knee—usually 32–128 for small encoders on modest GPUs, lower for long documents—and verify with the harness above rather than guessing. Anything beyond that knee is paying complexity for no throughput, and often worse latency. Measure against your actual model and host; the curve is cheap to plot and expensive to ignore.