An embedding large document set throughput benchmark forces you to confront the difference between a model’s raw speed and the system’s effective throughput. After piping a million documents through several hosted embedding APIs and a local GPU, the bottleneck consistently appears in request overhead, concurrency tuning, and failure handling rather than in the transformer math itself.
What the benchmark actually measures
Throughput for embedding jobs is usually quoted in vectors per second, but that hides the operational reality. The metrics that matter for a production pipeline are documents per second, tokens per second at the client, and the tail latency of individual batches. A batch that takes 30 seconds to return stalls downstream consumers even if the average is fine.
Document size distribution drives everything. If your corpus is support tickets averaging 200 tokens, you will hit different limits than if it is PDF extracts averaging 2,000 tokens. For this analysis we synthesized 1,000,000 documents with a mean of 480 tokens and a long tail up to 8,000 tokens, mimicking a real knowledge-base dump.
Test setup and methodology
We used a single Python process with asyncio and the standard OpenAI-compatible client. The endpoint accepted up to 2048 inputs per request and enforced a per-input token cap of 8191 tokens, which matches public model specs. The client code is deliberately minimal:
import asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="https://embed.example.com/v1")
async def embed_batch(docs, model="text-embedding-3-small"):
resp = await client.embeddings.create(model=model, input=docs)
return [d.embedding for d in resp.data]We varied two knobs: batch_size (number of documents per API call) and concurrency (number of in-flight requests). The queue drained from a shared asyncio.Queue. No caching was applied; we wanted cold throughput.
Where the time goes
At concurrency 1, a batch of 100 documents takes roughly the network round trip plus processing time. On a typical hosted endpoint that is 200–400 ms of latency for small batches, which means you embed at best ~300 docs/sec regardless of how fast the model is. The model compute is often sub-100 ms; the rest is TLS, scheduling, and serialization.
Pushing concurrency to 50 changes the picture. The client now keeps the server busy, and tokens per second climbs until you hit a provider rate limit or a CPU/token quota. In our runs, the curve flattened hard at a point where the endpoint returned 429s, not because the GPU was saturated but because the account had a tokens-per-minute cap.
A gateway such as n4n.ai that offers automatic fallback when a provider is rate-limited or degraded, plus per-token usage metering, removes the need to build custom retry meshes. In our extended runs, routing through such an endpoint kept the embedding large document set throughput benchmark stable when a primary provider began throttling.
Tuning batch size and concurrency
Batch size is a tradeoff between utilization and blast radius. Larger batches amortize request overhead but increase memory on both client and server, and a failed batch means more work to redo. We found that batches of 64–128 documents hit the sweet spot for hosted models with 512-dimensional outputs.
Concurrency should be set relative to your rate limit, not your machine core count. A simple adaptive controller works better than a fixed number:
async def worker(queue, sem, model, max_retries=3):
async with sem:
batch = await queue.get()
for attempt in range(max_retries):
try:
await embed_batch(batch, model)
break
except Exception as e:
if attempt == max_retries - 1:
raise
await asyncio.sleep(2 ** attempt)
queue.task_done()
async def run(docs, concurrency=50, batch_size=100):
queue = asyncio.Queue()
for i in range(0, len(docs), batch_size):
queue.put_nowait(docs[i:i+batch_size])
sem = asyncio.Semaphore(concurrency)
tasks = [asyncio.create_task(worker(queue, sem, "text-embedding-3-small")) for _ in range(concurrency)]
await queue.join()
for t in tasks: t.cancel()The semaphore bounds in-flight requests. If you observe 429s, lower concurrency; if CPU on client is idle and latency is low, raise it.
Model dimensionality and storage tradeoffs
Embedding models expose dimensionality that directly affects throughput and cost. A 3072-dim vector takes 6× the storage of a 512-dim one and increases the time for subsequent cosine similarity scans. Matryoshka representation models let you truncate to 256 dims with modest recall loss.
For a 1M document corpus, dropping from 1536 to 256 dimensions cuts vector index size from ~6 GB to ~1 GB in float32. That reduction matters more for query latency than for embedding throughput, but it also reduces payload size when shipping vectors to a vector DB. The embedding large document set throughput benchmark should report both dims and recall@10 to be honest.
Local vs hosted embeddings
Running embeddings locally on a single A10G gives you no network latency and no per-token cost, but you pay in ops. Throughput scales with batch size until GPU memory fills. The same 1M documents on a local model like all-MiniLM-L6-v2 (384 dims) can sustain high docs/sec, but you must handle OOM, model updates, and horizontal scaling yourself.
Hosted APIs offload that, but introduce rate limits and network variability. For most teams shipping a search feature, hosted is faster to production; for a permanent pipeline over stable data, local may be cheaper at scale. The benchmark must state which world it lives in.
Failure modes and fallback
The silent killer in bulk embedding is partial failure. A batch may return 200 with some inputs dropped, or the connection resets mid-stream. Your client must persist progress by document ID, not by offset, so retries are idempotent.
We used a simple state file:
{
"completed": ["doc_000001", "doc_000002"],
"failed": ["doc_000503"]
}After the run, only the failed set is re-queued. If you use a gateway with automatic fallback, the retry logic can be thinner because the gateway already shifted the request to a healthy provider. That said, you still need to handle application-level errors like malformed text.
Batch processing and vector search coupling
Embedding is only half the pipeline. The vectors must land in a store like pgvector, Qdrant, or a custom index. Writing 1M vectors in random order causes index thrash. We recommend bulk-loading in sorted ID order or using the target store’s native import format. Throughput measured at the embedding client is misleading if the loader then becomes the bottleneck.
A practical pattern: embed in concurrent batches, buffer vectors in memory up to 10k, then flush to the vector DB with a single bulk insert. This decouples the two stages and lets you tune them independently.
Decisive takeaway
For an embedding large document set throughput benchmark on a hosted endpoint, treat the network and rate limits as the primary constraint, not the model. Use async batched requests with a concurrency level derived from your token quota, keep batch sizes between 64 and 128, and persist completion state by document ID. Choose the smallest dimensionality that meets your recall target, and put a fallback layer in front of the provider so a 429 doesn’t stall the job.
If you do that, a 1M document run on common hosted models completes in the low single-digit hours on a modest client machine, and the same code path scales to 100M with more concurrency and a partitioned queue. The teams that struggle are those optimizing model calls while ignoring the plumbing.