Dimension size and its effect on embedding throughput

Embedding dimension size throughput is often treated as a simple knob: shrink the vector, speed up inference. That mental model breaks down with Matryoshka-trained models where the transformer backbone dominates compute and the final projection is cheap. The real leverage from reducing dimensions comes from memory footprint, batch packing, and downstream search latency—not from fewer FLOPs per token.

The compute reality: backbone dominates

Most engineers assume that cutting embedding dimension from 1536 to 512 halves the work. For a model that only does a final linear projection after a frozen transformer, that assumption is off by an order of magnitude.

Take a typical 300M-parameter embedding model. The transformer forward pass (attention, MLPs) accounts for >95% of MAC operations. The output head that maps hidden state to 1536 dims is a single matrix multiply of shape [hidden, dim]. If hidden=768, that head weighs 1.2M parameters for 1536 dims, 0.4M for 512. Compared to the 300M backbone, the head is noise.

What this means for token throughput

When you call an API with dimensions=512 on a model that supports Matryoshka truncation, the provider still runs the full forward pass. Measured token-per-second rates on the same hardware vary by single-digit percentages between 256 and 3072 dims. The embedding dimension size throughput in raw generation is therefore nearly flat.

# Conceptual benchmark loop—structure only, no fabricated results
from openai import OpenAI
client = OpenAI()

for dim in [256, 1024, 3072]:
    resp = client.embeddings.create(
        model="text-embedding-3-large",
        input=["sample text"] * 100,
        dimensions=dim
    )
    # time it; expect similar tokens/sec across dim

The takeaway: don’t expect a 3x speedup in embedding API calls by requesting smaller dims from the same model.

Where dimension size actually moves the needle

Dimension cuts the byte size of each vector linearly. A float32 1536-dim vector is 6KB; a 512-dim is 2KB. That difference dictates how many vectors you can stage in GPU memory during batching, and how fast your vector DB can scan.

Batch packing and VRAM

Embedding servers are often memory-bound on the output buffer, not compute-bound. If your service packs 8k sequences per batch, the output tensor is batch * seq * dim * 4 bytes. Dropping dim from 1536 to 512 shrinks that tensor 3x, letting you triple batch size before hitting the same memory ceiling. Effective system throughput (vectors embedded per second) scales with batch size until you saturate compute.

# Output buffer size for batch=8192, seq=1 (single vector per request)
dim=1536: 8192 * 1 * 1536 * 4 = 50.3 MB
dim=512:  8192 * 1 * 512  * 4 = 16.8 MB

That 3x memory relief often yields 2–2.5x more vectors/sec on the same GPU because you were previously batch-limited. This is the first place where embedding dimension size throughput improves as a system-level metric.

Downstream search latency is the bigger win

Vector similarity search cost scales with dimension. A brute-force cosine scan over 1M vectors at 1536 dim does 1.5B multiplies; at 512 dim it’s 0.5B. In ANN indexes (HNSW, IVF), lower dim reduces distance computation per node visit and shrinks cache lines.

If your RAG pipeline embeds at query time and then searches, the embedding call latency might be 20ms while the search at 1536 dim takes 40ms. Halving dim can cut search to 15ms. The end-to-end embedding dimension size throughput—meaning total system capacity—improves because the search side is no longer the bottleneck.

Memory of the index itself

HNSW graphs store raw vectors at each node. Cutting dim 3x cuts RAM cost 3x. On a 100M-vector corpus, that is the difference between 1.8TB and 600GB of RAM. For self-hosted Qdrant or Milvus, that decision determines whether you need 8 nodes or 24.

Tradeoffs: quality degradation is real

Smaller dimensions are not free. Matryoshka training mitigates but does not eliminate recall loss. On BEIR-style retrieval tasks, dropping from 1536 to 512 on text-embedding-3-small loses a few points of NDCG@10 in published evaluations; going to 256 loses more.

How to evaluate on your own data

Run a frozen recall test on your own corpus before committing.

import numpy as np

def recall_at_k(query_vec, doc_vecs, true_ids, k=10):
    sims = query_vec @ doc_vecs.T
    top = np.argsort(-sims)[:k]
    return len(set(top.tolist()) & set(true_ids)) / k

# Compare dim=1536 vs dim=512 embeddings of the same docs

If recall drop is <1% for your queries, ship the smaller dim. If it’s >5%, keep the larger. Never trust a provider’s marketing claim of “minimal loss”—measure against the queries that pay your salaries.

Batching strategy amplifies the memory effect

Concurrency masks latency but not memory. If you serve embeddings via a gateway, set max batch size based on dim. A dynamic scheduler that picks batch_size = f(dim) outperforms a static one.

Example batch policy

def max_batch(dim, vram_budget_mb=80):
    bytes_per_vec = dim * 4  # float32 assumption
    return int(vram_budget_mb * 1e6 / bytes_per_vec)

# dim=512 -> ~39k vectors; dim=1536 -> ~13k vectors

At dim=512, the scheduler admits three times the requests, raising aggregate embedding dimension size throughput without new hardware. The backbone compute is saturated either way, but you waste less cycles on memory-bound stalls.

Provider variability and routing

Different providers implement dimension truncation differently. Some truly train Matryoshka heads; others pad or truncate naive. The only way to know is to benchmark on your traffic.

Using a single OpenAI-compatible endpoint that fronts 240+ models—such as n4n.ai—lets you run identical client code while flipping model and dimensions across vendors. It honors routing directives and forwards cache-control, so you can A/B test a smaller-dim configuration on a fallback provider without rewriting your stack.

curl https://api.n4n.ai/v1/embeddings \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"provider/xenova-emb-512","input":"hello","dimensions":512}'

That portability turns dimension tuning from a code change into a config change. When a primary provider is rate-limited, automatic fallback preserves your benchmark methodology instead of silently swapping a model with different dim behavior.

Why memory bandwidth beats FLOPs

Modern GPUs compute matmuls faster than they can move data. The transformer backbone is memory-bandwidth bound: weights stream from HBM, activations stream back. Reducing output dim does not reduce weight traffic for the backbone. Therefore, token generation latency is anchored by HBM bandwidth, not by the tiny output projection.

If you switch to a fundamentally smaller model (e.g., BGE-base 768 vs BGE-large 1024), both backbone and dim shrink, and you will see real token throughput gains. But that is a model change, not a dimension parameter.

Honest cost analysis

Per-token pricing usually does not drop when you request fewer dims on the same model. You pay for the backbone compute regardless. So the cost saving from smaller dims is indirect: you fit more in batch, reduce vector DB storage, and cut search compute. If your bill is dominated by embedding API calls, dimension size alone won’t save money. If it’s dominated by vector DB RAM or query latency, it will.

Takeaway

Pick the smallest embedding dimension that holds retrieval quality on your data. Expect no magic speedup in raw embedding token generation from the same model—the backbone dominates. Capture the win through larger batches, lower memory, and faster similarity search. Measure recall on real queries, set batch size dynamically by dim, and treat dimension as a system-level throughput lever, not a model compute shortcut. The engineers who win here profile the whole pipeline, not just the embedding call.