When you’re weighing self-hosted embedding model throughput vs API options, the decision isn’t just about cost per token—it’s about tail latency, batching discipline, and whether your pipeline can tolerate a third-party outage. Most teams start with an API because it’s one curl call, then hit rate limits or per-token surprises at scale. Self-hosting flips the burden to GPU provisioning and model serving, but gives you predictable throughput and zero per-call fees.
Dimensions that actually matter
Capabilities
Self-hosted lets you run any open-weight model: BGE, E5, GTE, Nomic, or domain-specific fine-tunes. You control dimensionality, pooling, and normalization. API providers ship curated models (OpenAI text-embedding-3, Cohere embed-v3, Voyage) with documented multilingual and retrieval performance, but you can’t swap the architecture or inspect weights.
Price / cost model
APIs charge per token. OpenAI’s public list price is $0.02 per 1M tokens for text-embedding-3-small and $0.13 for text-embedding-3-large. Cohere and Voyage sit in the same order of magnitude. Self-hosting has zero marginal token cost, but you pay for GPU hours, storage, and engineering time. A single A10G on a cloud spot instance runs roughly $0.20–$0.40/hr, which becomes cheaper than API only after sustained volume.
Latency / throughput
This is where self-hosted embedding model throughput vs API options diverges hardest. A tuned Text Embeddings Inference (TEI) server on one A10G batches 512-token passages at often 500+ embeddings/sec, with sub-20ms preemption under load. API calls incur network round-trip (30–100ms regional) plus provider queueing; providers enforce per-minute token and request caps that throttle bursty backfills.
Ergonomics
APIs win on day one: an OpenAI-compatible client needs no infra. Self-hosting requires container orchestration, health checks, and a batching client. But self-hosted gives you a local endpoint that never 429s you during a reindex.
Ecosystem
Hugging Face supplies datasets, eval harnesses, and model hubs. API vendors supply SDKs, dashboards, and usage analytics. If you already use a gateway, you can route to multiple providers without rewriting calls.
Limits
APIs impose max input length (usually 8192 tokens) and rate tiers that need sales calls to raise. Self-hosted is bounded by your VRAM and the max batch tokens you configure; OOM kills are your own fault.
Self-hosted: what it really takes
Running an open model in production means using a serving stack built for transformers, not a naive Python loop. TEI is the pragmatic default.
docker run -p 8080:80 -v $PWD/data:/data \
ghcr.io/huggingface/text-embeddings-inference:latest \
--model-id BAAI/bge-large-en-v1.5 \
--max-batch-tokens 4096Client code is minimal:
import requests
resp = requests.post("http://localhost:8080/embed",
json={"inputs": ["chunk of text"], "normalize": True})
vec = resp.json()[0]["embedding"]Throughput scales near-linearly with batch size until you hit --max-batch-tokens. For a RAG backfill of 10M passages, self-hosted finishes in hours on one GPU; the same job via API may take days under free-tier limits or cost hundreds of dollars.
API options: managed embeddings
The call is boring in the best way:
from openai import OpenAI
client = OpenAI()
emb = client.embeddings.create(
model="text-embedding-3-small",
input=["chunk of text"]
).data[0].embeddingCohere and Voyage expose similar REST endpoints. If you want provider redundancy without racking GPUs, an OpenAI-compatible gateway such as n4n.ai fronts 240+ models with automatic fallback when a provider is rate-limited or degraded, and forwards cache-control hints so repeated corpora hit provider caches.
The trade-off is opacity. You don’t know if the provider silently changes the model version, and you’re bound by their compliance posture.
Head-to-head comparison
| Dimension | Self-hosted (TEI + open model) | OpenAI / Cohere / Voyage API | Gateway-fronted API |
|---|---|---|---|
| Model choice | Any open-weight | Fixed vendor models | Multiple via one endpoint |
| Cost at 100M tokens/mo | ~$150 GPU + ops | $2–$13 (API list) | API cost + small gateway margin |
| Throughput | 500+ emb/s on A10G, scales with HW | Bounded by rate limit (tens of k tok/min) | Same as underlying API |
| Tail latency | Sub-50ms local | 30–150ms network + queue | +1 hop, similar |
| Ops burden | High (deploy, patch, scale) | None | Low (config only) |
| Data residency | Your VPC | Vendor cloud | Vendor cloud + gateway |
| Rate limit pain | None (self-throttle) | Real, especially backfills | Mitigated by fallback |
Which to choose
Early-stage prototype or low-volume app. Use the API. The per-token cost is trivial, and you ship in an afternoon. Don’t overestimate your embedding volume—most side projects never cross 10M tokens.
High-volume RAG or constant reindexing. Self-host. Once you’re embedding millions of documents monthly, GPU cost beats API line items, and you remove a network dependency from your search path. Use TEI, pin the model revision, and batch aggressively.
Regulated or air-gapped data. Self-host in your own VPC. No API vendor can promise what a local container already guarantees.
Multi-provider experimentation. If you need to A/B Cohere vs OpenAI without refactoring, a gateway that honors client routing directives saves weeks. This is the only case where the extra hop pays for itself in optionality.
Bursty backfills with SLA. API alone will 429 you. Either self-host for the backfill window or use a gateway with fallback and per-token metering so finance sees the spike. Self-hosted batch jobs on spot GPUs are usually cheapest.
The self-hosted embedding model throughput vs API options debate resolves to ownership: APIs sell convenience and opacity; self-hosting sells predictability and toil. Pick the side that matches your volume and your on-call roster.