Voyage AI embeddings: throughput at large batch sizes

Voyage AI embeddings throughput scales sharply with batch size at the low end, then plateaus long before you reach the provider’s request limits. Batch size of 1 wastes most of the round-trip on fixed overhead; batch size of 128 amortizes that overhead across hundreds of texts and typically delivers the best tokens-per-second you can get from a single request stream. Pushing further with giant batches rarely helps and often hurts tail latency, so the engineering win comes from pairing the maximum sane batch size with concurrent request streams.

The fixed cost of an embedding request

Every embedding call pays a tax independent of payload: TLS handshake (or connection reuse), HTTP headers, JSON parsing, authentication check, and scheduler time on the provider side. For a single short sentence, that fixed cost can exceed the compute time to actually embed the text. Voyage’s inference servers still need to allocate a batch on the GPU, and a batch of one is the worst case for utilization.

Consider the naive loop:

import openai
client = openai.Client(base_url="https://api.voyageai.com/v1", api_key="KEY")

for text in texts:
    client.embeddings.create(model="voyage-2", input=text)

Each call serializes, sends, waits, and deserializes. Even on a warm connection, you’re bounded by RTT times the number of items. Batching collapses N RTTs into one. The fixed cost does not disappear, but it is divided by the batch size.

Voyage’s documented limits and token math

Voyage caps the number of inputs per request and the token count per input. For voyage-2 the per-request limit is 128 inputs; each input may be up to 4096 tokens. Other models follow similar shapes. That means a single request can carry up to roughly 128 × 4096 = 524,288 tokens, but in practice most embedding workloads use short chunks of 256–512 tokens.

If you send 1,000 documents of 500 tokens each, a batch size of 128 packs them into 8 requests. A batch size of 8 would need 125 requests. The math is simple, but the throughput delta is not linear because the provider also parallelizes internally up to a point.

BATCH = 128
for i in range(0, len(texts), BATCH):
    chunk = texts[i:i+BATCH]
    client.embeddings.create(model="voyage-2", input=chunk)

Output dimensionality and payload size

Voyage-2 returns 1024-dimensional vectors; voyage-large-2 returns 1536. The response body scales as batch_size × dims × 4 bytes for float32. A 128-batch of voyage-large-2 ships back about 128 × 1536 × 4 ≈ 786 KB. That is not huge, but it is not free on a saturated link. Smaller batches reduce per-response size and can lower time-to-first-byte for the caller, even if total throughput is lower.

Where the curve bends

Throughput (embeddings/sec) climbs as batch size increases because GPU kernels favor large matrices. But the provider’s GPU memory and inference server cap how much parallel work a single request can trigger. Once the batch saturates the serving stack, adding more inputs to the same request just queues them internally; you get no extra tokens/sec.

Empirically, for Voyage’s standard models, the knee sits near the documented max batch. Going from 1 to 32 yields a 10–20× throughput jump. Going from 32 to 128 yields another 2–3×. Going from 128 to 256 (if allowed) yields <1.2× and may increase per-request timeout risk. We are not quoting exact benchmarks; the relative shape is what matters for design.

Rate limits and batch efficiency

Providers enforce requests-per-minute (RPM) caps. A workload sending 1,000 items at batch 1 consumes 1,000 requests; at batch 128 it consumes 8. If your RPM quota is 300, the small-batch approach gets throttled immediately while the large-batch approach runs comfortably. Voyage AI embeddings throughput is therefore also a function of how efficiently you stay under RPM by maximizing per-request item count.

Concurrency, not bigger batches

After you hit the batch knee, the only way to push more volume is to send multiple batches in flight. HTTP/2 or multiple connections allow this. Use an async client:

import asyncio, openai
client = openai.AsyncClient(base_url="https://api.voyageai.com/v1", api_key="KEY")

async def send(chunk):
    return await client.embeddings.create(model="voyage-2", input=chunk)

async def run(texts, batch=128, conc=4):
    sem = asyncio.Semaphore(conc)
    async def bounded(ch):
        async with sem:
            return await send(ch)
    tasks = [bounded(texts[i:i+batch]) for i in range(0, len(texts), batch)]
    return await asyncio.gather(*tasks)

Four concurrent streams of 128-sized batches typically double throughput again versus a single stream, without the latency blow-up of a 512-size batch.

Connection pooling

The async client reuses a connection pool. If you self-host a worker that embeds continuously, set a sensible pool size matching your concurrency. Too few connections and you serialize; too many and you hammer the provider and trigger jitter.

Tradeoffs: latency, memory, and failure blast radius

Large batches reduce per-item overhead but increase worst-case latency. A batch of 128 returns only after all 128 are embedded; if one input is a pathological 4000-token essay, the whole batch waits. Memory on your side grows: you hold all inputs and all output vectors. At concurrency 10 × batch 128 × 1536 dims you buffer roughly 8 MB of vectors in memory—manageable, but worth noting.

Retries are uglier. If a 128-item request fails at 90% done, you resend all 128. With smaller batches, the blast radius is smaller. I prefer batch size 128 and concurrency 3–5; if error rates climb, drop batch to 64.

Gateway routing and fallback

When you call Voyage through an OpenAI-compatible gateway such as n4n.ai, the batching decision remains entirely client-side. The gateway’s automatic fallback triggers only when the provider returns rate-limit or degradation errors; it will not repack your oversized batch or split it. You still must honor Voyage’s input limits. The gateway does forward provider cache-control hints, but embeddings are deterministic and rarely cached, so that hint is inert here.

Production pattern

A robust pipeline looks like:

1. Chunk documents to <= 512 tokens.
2. Group into batches of 128.
3. Dispatch with asyncio, limit concurrency to 4.
4. On exception, retry the batch with exponential backoff; after two fails, halve batch size.
5. Meter tokens via response usage field (or gateway per-token metering) to track cost.

async def embed_pipeline(texts):
    batch, conc = 128, 4
    while True:
        try:
            return await run(texts, batch, conc)
        except openai.APIError:
            if batch <= 16:
                raise
            batch //= 2

This pattern keeps you near the throughput knee, respects provider limits, and degrades gracefully under instability.

Takeaway

Use the largest batch size Voyage allows as your default—usually 128—because Voyage AI embeddings throughput gains are front-loaded and flatten afterward. Do not try to circumvent the limit with bigger batches; add concurrent request streams instead. That combination maximizes tokens-per-second while keeping tail latency and retry cost acceptable for production systems.