Jina Embeddings v3 throughput across batch sizes

The Jina Embeddings v3 throughput benchmark reveals a familiar pattern for transformer encoders: batch size is the single biggest lever for tokens processed per second. This analysis digs into why that holds, how to find the knee in the curve, and what it means for production embedding pipelines serving multilingual RAG or search at scale.

Why batch size dominates embedding throughput

Embedding models are inference-only transformers that run a single forward pass per input. Unlike autoregressive decoders, they have no KV-cache growth per token and no sequential sampling step. The compute per sequence is fixed by token count and layer operations, which makes the workload predictable but also unforgiving of inefficient scheduling.

That predictability is exactly why batching works so well. A single short sequence underutilizes the GPU because the bottleneck is streaming weights from HBM, not performing matmuls. Each kernel launch also carries fixed overhead. If you issue one request at a time, you pay that overhead repeatedly and leave compute units starved.

Jina Embeddings v3 is a 570M-parameter multilingual encoder with an 8192-token context, Matryoshka representation learning, and optional late-interaction (ColBERT-style) token vectors. The architecture is a standard 24-layer BERT-like transformer with hidden size 1024 and intermediate size 4096. None of that changes the batching principle; it only shifts where the wall appears.

The memory-bandwidth wall

A useful mental model: throughput (tokens/sec) ≈ (available memory bandwidth) / (bytes per param × param count) × batch utilization. In fp16, 570M params is roughly 1.14 GB. An A100 80GB delivers about 2 TB/s of HBM bandwidth. The theoretical ceiling is high, but real kernels typically hit 40–60% of the roofline due to imperfect tiling and activation spills.

As batch size grows, utilization climbs because weight fetches are amortized across many sequences. Activation memory per sequence at seqlen 512 is small—on the order of a few MB—so an 80GB card can hold hundreds of sequences. The knee in the curve shows up when adding more sequences no longer improves tokens/sec because memory bandwidth or kernel occupancy is saturated.

In a Jina Embeddings v3 throughput benchmark, you typically see near-linear tokens/sec gains from batch 1 to 16, then sublinear scaling to 64, then a flat plateau. Past that point you are only increasing per-batch latency without throughput benefit.

Late interaction changes the math

Jina’s late-interaction mode returns a tensor of shape (seq_len, 1024) per document instead of a single pooled (1024,) vector. That multiplies output bytes by the sequence length. Network egress and client-side deserialization become the limiting factor, not GPU compute.

Most semantic search only needs the pooled sentence embedding. Disable late interaction unless you are explicitly building a ColBERT retriever. The Matryoshka property lets you slice the 1024-dim vector to 256 or 128 dims with small accuracy drops on many benchmarks. Smaller vectors reduce post-processing, storage, and bandwidth costs.

from openai import OpenAI

client = OpenAI(base_url="https://api.jina.ai/v1", api_key="...")

# Batch of 32 documents, default 1024-dim pooled embedding
texts = ["document text here"] * 32
resp = client.embeddings.create(
    model="jina-embeddings-v3",
    input=texts,
)
# resp.data[i].embedding is length 1024
# late interaction would return an additional token-level array per item

If you do enable late interaction, expect sequences/sec to drop by several multiples at equal batch size because payload size balloons with sequence length.

Finding the optimal batch size in practice

Optimal batch size is context-dependent. Offline backfills of a vector database should maximize batch size until per-batch latency breaches the job’s deadline. Online serving must micro-batch across user requests within a short time window (10–50 ms) to hit cost targets.

A minimal load test harness helps you locate the knee on your own hardware or provider:

# pip install openai
python bench.py --batch-sizes 1,8,16,32,64 --seq-len 512 --requests 200

import time, argparse
from openai import OpenAI

def bench(batch_sizes, seq_len, requests):
    client = OpenAI(base_url="https://api.jina.ai/v1", api_key="...")
    doc = "word " * seq_len
    for b in batch_sizes:
        start = time.time()
        for _ in range(requests):
            client.embeddings.create(
                model="jina-embeddings-v3",
                input=[doc] * b
            )
        elapsed = time.time() - start
        tok = requests * b * seq_len
        print(f"batch={b} tok/s={tok/elapsed:.0f} lat={elapsed/requests*1000:.1f}ms")

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--batch-sizes", default="1,8,16,32")
    ap.add_argument("--seq-len", type=int, default=512)
    ap.add_argument("--requests", type=int, default=100)
    args = ap.parse_args()
    bench([int(x) for x in args.batch_sizes.split(",")], args.seq_len, args.requests)

Run this against your deployment. The absolute numbers will vary, but the curve shape is stable: a clear knee, then plateau.

Gateway considerations

When you front embedding calls with an OpenAI-compatible gateway such as n4n.ai, the same client code works while the gateway handles provider fallback and per-token metering. That lets you shift traffic to a secondary region when your primary Jina endpoint is rate-limited, without rewriting batching logic or measurement harnesses.

Tradeoffs: latency vs throughput

Larger batches always raise per-request p99 latency. A batch of 64 at seqlen 512 might take 200 ms end-to-end; a batch of 1 takes 15 ms. If your SLA is 50 ms, you cannot blindly max batch size.

Micro-batching mitigates this: accumulate requests for 20 ms, then send as one batch. You trade a small fixed delay for 4–8× throughput. This is how most embedding APIs stay profitable. A simple asyncio collector can do it:

import asyncio, time
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://api.jina.ai/v1", api_key="...")

async def microbatch(inputs, window=0.02):
    tasks = []
    start = time.time()
    while time.time() - start < window and len(tasks) < 64:
        if inputs:
            tasks.append(client.embeddings.create(
                model="jina-embeddings-v3", input=inputs.pop(0)
            ))
    return await asyncio.gather(*tasks)

Another tradeoff is sequence length. Jina supports 8192 tokens, but long docs crush throughput because attention memory is O(n²) in activation size. Truncate to 512 or 1024 unless you truly need the whole document embedded. The Jina Embeddings v3 throughput benchmark at 8k context will look an order of magnitude worse than at 512 for the same batch size.

Capacity planning with batch size

Suppose you need to embed 100M documents of avg 300 tokens. At batch 32 and a realistic 20k tok/s on a single A100-class GPU, that is 100M×300 / 20k ≈ 1.5M seconds ≈ 17 days on one card. Bumping to batch 64 might get 30k tok/s, cutting to 11 days. The marginal gain from 32→64 is rarely worth the latency hit for online systems, but for offline jobs it is free speed.

Dimension reduction via Matryoshka from 1024 to 256 cuts vector store size 4× and speeds similarity search. That indirect throughput win matters more at query time than at embed time.

Takeaway

For nearly all production deployments, set batch size between 16 and 32 for offline jobs, and use 10–30 ms micro-batching for online serving. Disable late interaction unless you are building ColBERT retrieval. Truncate to 512 tokens and shrink dimensions to 256 via Matryoshka if accuracy allows. The Jina Embeddings v3 throughput benchmark confirms that batching is the cheapest performance win available—running it at batch 1 in production is leaving money and GPU cycles on the table.