BGE-M3 vs OpenAI embeddings: tokens per second benchmarked

The argument around BGE-M3 vs OpenAI embeddings throughput is really an argument about where the bottleneck lives. Run BGE-M3 on your own GPU and embedding becomes a batch inference problem you control; call OpenAI’s API and it becomes a network latency and rate-limit problem governed by someone else’s capacity. This post puts both options side by side across capabilities, cost, latency, ergonomics, and ecosystem so you can choose based on measured tradeoffs rather than hype.

Capabilities

BGE-M3 is a single model that produces three embedding types at once: dense (1024-dim), sparse (lexical weights), and multi-vector (ColBERT-style). That makes it a Swiss Army knife for hybrid retrieval—you can feed the same forward pass into a dense ANN index and a BM25-style sparse index without running separate models. It handles 100+ languages and accepts up to 8192 tokens per input.

OpenAI’s current embedding lineup (text-embedding-3-small, text-embedding-3-large) is dense-only. Dimensions are 1536 and 3072 respectively, with a 8191-token limit. You get strong English-centric multilingual quality and easy dimensionality reduction via the dimensions parameter, but no native sparse or late-interaction vectors. If your retrieval stack needs lexical signals, you have to bolt on a separate tokenizer or use a third-party sparse encoder.

# OpenAI dense-only
from openai import OpenAI
client = OpenAI()
resp = client.embeddings.create(
    model="text-embedding-3-small",
    input=["query text", "doc text"],
    dimensions=512
)

# BGE-M3 multi-function
from FlagEmbedding import BGEM3FlagModel
model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)
out = model.encode(["query text", "doc text"],
                   return_dense=True, return_sparse=True, return_colbert_vecs=False)

Price and cost model

OpenAI charges per token. Public list price for text-embedding-3-small is $0.02 per 1M tokens and text-embedding-3-large is $0.13 per 1M tokens. At RAG scale—say 500M chunks embedded per month—that is $10–$65 on the small model, plus the cost of re-embedding on schema changes.

BGE-M3 is MIT-licensed and free to download. Your cost is infrastructure: a single A10G (24GB) instance on a typical cloud runs about $0.30–$0.80/hr depending on region and commitment. If you embed 500M tokens in a batch job that takes 10 GPU-hours, you pay $3–$8 for compute. The catch is you pay that cost whether or not you use the GPU continuously; idle clusters are pure waste. Amortize the GPU across multiple services or use spot instances to close the gap.

Latency and throughput

This is where the BGE-M3 vs OpenAI embeddings throughput comparison gets concrete. Local inference throughput scales with batch size and GPU memory. With a tuned serving stack (Text Embeddings Inference or ONNX Runtime), BGE-M3 on an A10G sustains throughput in the tens of thousands of tokens per second per GPU when requests are batched. Tail latency at batch=1 is sub-10ms on the GPU plus minimal Python overhead.

OpenAI’s API hides the hardware but exposes its own ceiling. Each request carries ~20–50ms of network and queueing latency even for tiny payloads. You can pack up to 2048 inputs per request, but the effective tokens/sec you observe is bounded by your tier’s rate limit (tokens per minute) and concurrency. In practice, a single client thread sees a few thousand tokens/sec; spreading across many async workers gets you closer to the provider cap but never eliminates per-request tax.

Measuring local throughput

When you serve BGE-M3 with TEI, the endpoint accepts a JSON array and returns embeddings. A realistic throughput test batches 64 sequences of 512 tokens. On an A10G, the GPU compute time per batch is tiny; the limiter is Python client concurrency and HTTP serialization. We’ve seen local setups sustain an order of magnitude more tokens per second than a single-threaded API client can pull from OpenAI, simply because there is no TLS and round-trip penalty per batch.

# quick local load test against TEI
for i in {1..100}; do
  curl -s -X POST http://localhost:8080/embed \
    -H 'Content-Type: application/json' \
    -d '{"inputs":["repeat text here for 512 tokens", "second doc"], "normalize":true}' >/dev/null &
done

The BGE-M3 vs OpenAI embeddings throughput gap narrows only when you cannot batch—e.g., online single-query embedding at p99. There, OpenAI’s global fleet may actually deliver lower latency from a nearby edge than a GPU sitting idle in a cheap region.

Below is a minimal async benchmark harness that measures effective client-side throughput for either endpoint:

import asyncio, time, openai

async def bench_openai(texts, concurrency=32):
    client = openai.AsyncOpenAI()
    sem = asyncio.Semaphore(concurrency)
    async def worker(batch):
        async with sem:
            await client.embeddings.create(model="text-embedding-3-small", input=batch)
    t0 = time.monotonic()
    await asyncio.gather(*[worker(texts[i:i+128]) for i in range(0, len(texts), 128)])
    return len(texts) / (time.monotonic() - t0)

# For BGE-M3, point the same logic at your local TEI endpoint:
# POST http://localhost:8080/embed with {"inputs": batch}

The key takeaway: if you can batch, local BGE-M3 wins on raw tokens/sec by an order of magnitude. If you need sporadic single-doc embeddings with zero ops, OpenAI’s flat latency is simpler to reason about.

Ergonomics

OpenAI’s embedding API is one import and one call. No model weights, no CUDA version mismatches, no worrying about fp16 vs bf16. You scale by paying more, not by hiring a platform team.

BGE-M3 requires you to pick a serving path. The quickest local route is pip install FlagEmbedding and load the model, but that blocks a GPU in your Python process. For production you’ll likely run Text Embeddings Inference:

docker run -p 8080:80 \
  -e MODEL_ID=BAAI/bge-m3 \
  -e MAX_BATCH_SIZE=32 \
  ghcr.io/huggingface/text-embeddings-inference:latest

Then you call it like an OpenAI endpoint with a thin adapter. If you route through an OpenAI-compatible gateway such as n4n.ai, the same client code can target either provider by changing the model string, and you keep per-token metering without custom instrumentation.

Ecosystem

Both models are first-class in LangChain and LlamaIndex. BGE-M3 has deeper integration with open-source vector DBs that support hybrid search (Qdrant, Milvus) because it emits sparse vectors natively. OpenAI embeddings are the default in countless SaaS retrieval templates and have the largest community of copy-paste snippets.

For sparse retrieval, you’ll often see BGE-M3 paired with a custom tokenizer index; OpenAI users typically fall back to a separate BM25 library. That split shapes your codebase more than the raw vector math does.

Limits

OpenAI limits: rate tiers, no sparse/multi-vector, data leaves your boundary, and you can’t inspect the model for drift. BGE-M3 limits: you own uptime, GPU memory (the full model in fp16 needs ~4–6GB, but ColBERT vectors explode storage), and you must handle version upgrades yourself.

A subtle BGE-M3 limit is multi-vector storage cost. Returning ColBERT-style tokens multiplies embedding size by sequence length; at 8192 tokens that’s millions of floats per doc. Most teams disable return_colbert_vecs in production and keep only dense+sparse.

Head-to-head summary

Which to choose

Prototype or low-volume SaaS: Use OpenAI embeddings. The per-token cost is negligible at <100M tokens/month, and you ship in an afternoon. The BGE-M3 vs OpenAI embeddings throughput gap does not matter when you’re embedding a few docs per request.

Large-scale RAG or indexing pipeline: Self-host BGE-M3. Once you cross ~1B tokens/year, GPU amortization beats API bills, and local batching gives you predictable latency. Use TEI or ONNX and disable ColBERT vectors unless you truly need late interaction.

Privacy-regulated data: BGE-M3 (or any self-hosted model) keeps text on your hardware. OpenAI’s API transmits payloads off-prem; even with zero-retention flags, that’s a compliance conversation.

Hybrid retrieval without extra services: BGE-M3 is the only one of the two that gives you sparse weights from the same forward pass. If you want lexical + semantic in one call, it’s the default.

Elastic, unpredictable burst load: OpenAI wins on elastic scale—no cluster to autoscale, no cold GPU. If your traffic spikes 100x hourly, the API absorbs it (up to your tier); your self-hosted cluster will either over-provision or throttle.

Pick based on where your tokens live and who answers for uptime. The throughput numbers are real, but the operational weight is what ships or sinks the project.