Nomic Embed vs text-embedding-3-small: throughput benchmarks

When you’re weighing Nomic Embed vs text-embedding-3-small throughput for a production retrieval pipeline, the headline numbers hide more than they reveal. Both target semantic search, clustering, and classification, but their operational profiles diverge sharply once you move past accuracy leaderboards and start shipping traffic.

Capabilities

Dimensionality and Matryoshka support

Nomic Embed Text (v1.5) ships a fixed 768-dimensional vector. You can post-process with PCA or truncate, but the model is not natively matryoshka. OpenAI’s text-embedding-3-small defaults to 1536 dimensions and explicitly supports the dimensions parameter, letting you request 512 or 256 vectors without retraining. That flexibility directly changes your vector DB memory footprint and approximate nearest neighbor search cost.

Context length and language coverage

Nomic trains on an English-and-code-heavy mix with some multilingual exposure; the context window is 8192 tokens. text-embedding-3-small has an 8191-token limit and performs best on English, with weaker but usable non-English recall. If your corpus is German support tickets or Japanese logs, run a small eval before trusting either. On MTEB-style English tasks, Nomic sits close to OpenAI’s older ada-002 and slightly behind the 3-small in aggregate, but the gap is use-case dependent.

Price and Cost Model

OpenAI charges $0.02 per 1M tokens for text-embedding-3-small (public pricing as of 2024). You pay per call, with zero infrastructure overhead. Nomic Embed is open-weight under Apache-2.0: you bring the GPU. A single A10G on spot instances costs pennies per hour and can embed millions of short documents per dollar if batch utilization stays high.

The break-even point depends on volume and sequence length. Above roughly 100M tokens per month, self-hosting usually wins on pure cost. Below that, the engineering time to operate the service often outweighs the API line item. Don’t forget hidden costs: larger OpenAI dimensions inflate vector storage and egress fees in your database.

Latency and Throughput

This is where Nomic Embed vs text-embedding-3-small throughput becomes a hardware-versus-network argument.

Self-hosted Nomic

Running the model via sentence-transformers or ONNX runtime locally, you control batch size and concurrency. On a modest GPU, throughput scales near-linearly with batch up to VRAM limits. Typical observation: sub-10ms per document at batch 1, dropping to sub-millisecond per doc at batch 64. There is no network round-trip and no provider rate limit. Your bottleneck is your own queue and preprocessing.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5")
# batch inference keeps the GPU saturated
embs = model.encode(corpus, batch_size=64, convert_to_numpy=True)

API-bound OpenAI

You ship text over TLS and wait for a managed worker. Median latency for a single short query is 20–50ms from US-east; batching helps but you still pay per token and hit requests-per-minute caps. Default tiers allow on the order of 10k requests/minute, but large batches eat token quotas fast. Throughput plateaus when you saturate your client’s connection pool, not their silicon.

from openai import OpenAI
client = OpenAI()
resp = client.embeddings.create(
    model="text-embedding-3-small",
    input=corpus,
    dimensions=512
)

If you route both models through a single OpenAI-compatible gateway such as n4n.ai, you get per-token metering and automatic fallback when a provider is degraded, but the underlying throughput constraints remain tied to the execution path.

Batch size effects

Nomic benefits dramatically from large batches; OpenAI’s API accepts arrays up to 2048 inputs but each counts toward token limits. For bulk backfills, self-hosted wins on raw cost and avoids network jitter. For sporadic real-time lookups, API latency is acceptable and frees you from capacity planning.

Measuring it yourself

Don’t trust vendor claims. Stand up a loop:

import time, numpy as np
def bench(model_fn, texts, bs=32, n=100):
    t0 = time.time()
    for i in range(0, len(texts), bs):
        model_fn(texts[i:i+bs])
    return n * len(texts) / (time.time() - t0)

Run against your own document distribution. Short social posts behave differently than long PDF chunks.

Ergonomics and Integration

OpenAI’s endpoint is ubiquitous: every LangChain, LlamaIndex, and Pinecone tutorial assumes it. Nomic requires you to stand up a service or call a local library. Containerizing the model behind an OpenAI-compatible stub takes an afternoon but adds a deployment surface.

If you already run inference clusters, Nomic drops in cleanly. If you’re a solo developer or a small team without GPU ops, the API key is lighter. The client code is nearly identical once you hide the model behind an abstraction.

Ecosystem and Tooling

Nomic releases weights on HuggingFace with a permissive license. You get ONNX, GGUF, and community quantizations that run on CPU or Apple Silicon. OpenAI gives you zero weights but maximal third-party support: vendor SDKs, proxy layers, and observability hooks. For regulated data, Nomic’s self-host eliminates exfiltration risk entirely; OpenAI requires a DPA and trust.

Limits and Failure Modes

Nomic’s limit is operational: you patch CUDA, scale pods, handle OOM, and monitor GPU utilization. OpenAI’s limit is policy: they can deprecate, rate-limit, or change dimensions with notice. Also, text-embedding-3-small’s matryoshka truncation slightly hurts recall at 256-dim; Nomic at 768 is stable. Both choke on documents exceeding 8k tokens—you must chunk before embedding.

Comparison Table

Which to Choose

High-volume RAG or batch indexing

Self-host Nomic. Once you cross millions of docs, API spend and latency variance hurt. Own the stack, tune batch size, and use spot GPUs.

Low-volume prototypes

Use text-embedding-3-small. Zero infra, instant integration, easy swap later when requirements firm up.

Multilingual or long-doc heavy

Test both on your eval set. Nomic’s 768-dim often holds recall on English; OpenAI’s larger dim helps cross-lingual but verify with real queries.

Cost-sensitive scheduled jobs

Nomic on spot GPUs beats per-token billing. Write a cron that embeds nightly and shuts the instance down.

Real-time user queries with sporadic traffic

OpenAI API avoids keeping a GPU warm for nothing. Pay per use and let someone else handle uptime.

Pick based on who runs the metal and how predictable your volume is, not just the MTEB bar chart.