Benchmarking embedding latency for real-time RAG search

Embedding latency RAG search is the silent killer of interactive retrieval experiences. While teams obsess over LLM token throughput, the milliseconds spent turning a user query into a vector before any similarity scan often dictate whether a search feels instant or sluggish.

Why embedding latency RAG search matters more than you think

In a real-time RAG system the user is blocked until the entire retrieve-then-generate cycle finishes. The generation step can be streamed to the UI, masking part of its cost. The retrieval step cannot. A query must be embedded before the vector database can return candidates, and that embedding call sits squarely on the critical path.

If your product targets a sub-200 ms response feel, and your vector search plus reranking takes 30 ms, you have roughly 100 ms left for embedding and network overhead. A single call to a hosted embedding API from a different region can consume that budget entirely. This is why embedding latency RAG search deserves the same profiling rigor you apply to database queries.

The anatomy of a real-time query path

A typical path looks like this:

1. Client sends POST /search with free-text query.
2. Server embeds the query (local model or remote API).
3. Vector DB executes ANN search.
4. Optional reranker scores top-k.
5. LLM synthesizes answer from passages.

Steps 2–4 are sequential. Here is a minimal Python snippet showing the naive synchronous version:

import openai

def retrieve(query: str, top_k: int = 5):
    # Step 2: embed via cloud API
    resp = openai.Embedding.create(
        model="text-embedding-3-small",
        input=query
    )
    vec = resp["data"][0]["embedding"]
    # Step 3: pseudo-code vector search
    hits = vector_db.search(vec, top_k=top_k)
    return hits

The openai.Embedding.create call blocks the event loop if run synchronously. Even under async, the round trip latency is unchanged.

Methodology: what we measured and how

We built a small harness that fires concurrent queries and records per-call latency distributions. The point is not to publish absolute numbers—hardware and region skew results—but to show the shape of the problem. The harness runs against two backends:

• A local sentence-transformers model (all-MiniLM-L6-v2) loaded on the same process.
• A hosted embedding endpoint over HTTPS.

import time, asyncio, statistics
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

async def timed_embed(texts, backend):
    start = time.perf_counter()
    if backend == "local":
        model.encode(texts)
    else:
        # assume async client wrapped
        await cloud_embed(texts)
    return time.perf_counter() - start

# Run 1000 iterations, collect samples, compute p50/p99

The key observation from running this: local inference latency is dominated by compute and stays flat under modest concurrency, while cloud latency is dominated by network and TLS handshake, and exhibits a long tail when the provider throttles.

Cloud embeddings: convenience at a latency cost

Hosted embeddings remove the burden of model hosting, versioning, and GPU provisioning. They typically expose an OpenAI-compatible interface, making integration trivial. The tradeoff is that every query pays a serialization, network, and possibly queuing cost.

An inference gateway such as n4n.ai can mask provider degradation with automatic fallback when a primary embedding service is rate-limited, but the fundamental network floor remains: you cannot make a packet cross a continent faster than physics allows.

If your application server runs in us-east-1 and your embedding API is also in us-east-1, you might see low tens of milliseconds. If you are in eu-central-1 calling a US endpoint, double it. For real-time RAG search, that variance is what breaks SLOs.

Local models: fast but operationally heavier

Running a small transformer locally flips the latency profile. A 22M-parameter model like all-MiniLM-L6-v2 processes a single sentence in low single-digit milliseconds on a modern CPU, and sub-millisecond on a GPU. Because the model lives in the same memory space as your API, there is no network hop.

The cost is operational: you must load the model (typically 80–200 MB), manage memory, and handle version upgrades. For many teams, a sidecar container or a dedicated embedding microservice solves this cleanly.

# Launch a local embedding service with sentence-transformers
docker run -p 8080:80 -e MODEL_ID=all-MiniLM-L6-v2 \
  ghcr.io/huggingface/text-embeddings-inference:latest

This gives you a local HTTP endpoint that mimics the cloud shape without the wide-area latency.

Batching and concurrency traps

Embedding APIs are usually priced per token and optimized for batch jobs: send 100 documents, get 100 vectors. Real-time query embedding is the opposite—one short string per request. If you naively batch queries from different users to save cost, you introduce queueing delay. A micro-batch of 10 queries at 5 ms each becomes a 50 ms p99 for the unlucky last query.

If you stay local, batching is still useful but for throughput, not latency. Keep the online path synchronous per request, and use a separate worker for bulk corpus ingestion.

Caching and pre-embedding

The cheapest embedding is the one you never compute. Query strings repeat: “refund policy”, “how to reset password”. A simple LRU cache keyed by normalized query text eliminates repeated embedding calls entirely. For multilingual scenarios, normalize casing and strip punctuation before cache lookup.

On the corpus side, embeddings are precomputed at index time—that is table stakes. The less obvious win is a semantic cache: store the final retrieved passage IDs for a query embedding, so a near-duplicate query can skip vector search too.

Quality vs speed tradeoffs

Small local models are fast but may underperform larger cloud models on niche domains. all-MiniLM-L6-v2 is trained on general web text; a legal RAG system may see recall drop. The mitigation is hybrid search: combine the fast embedding with lexical BM25 scores. The embedding still must be computed, but you can afford a smaller model because lexical signals cover its blind spots.

{
  "retrieval": {
    "vector_weight": 0.6,
    "bm25_weight": 0.4,
    "embedding_model": "local:all-MiniLM-L6-v2"
  }
}

This config keeps embedding latency low while preserving result quality.

Reference implementation

Below is a compact FastAPI route that uses a local model with a cloud fallback. It measures embedding time and tags it for metrics.

from fastapi import FastAPI
from sentence_transformers import SentenceTransformer
import openai, time

app = FastAPI()
local_model = SentenceTransformer("all-MiniLM-L6-v2")

@app.post("/embed")
async def embed(q: str):
    t0 = time.perf_counter()
    try:
        vec = local_model.encode(q).tolist()
        backend = "local"
    except Exception:
        resp = openai.Embedding.create(model="text-embedding-3-small", input=q)
        vec = resp["data"][0]["embedding"]
        backend = "cloud"
    elapsed = (time.perf_counter() - t0) * 1000
    # export to metrics: embedding_latency_ms{backend=...}
    return {"vec": vec[:3], "ms": elapsed, "backend": backend}

This pattern lets you default to low-latency local inference and only pay the network cost when the local process is unhealthy.

Decisive takeaway

Treat embedding latency RAG search as a primary SLO, not an afterthought. For interactive systems, deploy a small local embedding model on the same network boundary as your API, cache aggressively, and reserve cloud embeddings for fallback or bulk jobs. Measure p99 under production concurrency, not average latency in a notebook. If you need cross-lingual or state-of-the-art recall, isolate those queries and route them to larger models via a gateway with fallback—but never let a network call block your hot path by default. The fastest embedding is the one that never leaves the box.