Measuring latency overhead of query rewriting in RAG

Query rewriting latency RAG systems is rarely measured before it ships, yet it can add 200–800 ms to every user request depending on model size and call pattern. The thesis here is simple: rewriting is a tax you pay twice—once for the rewrite call, once for the downstream retrieval and generation—and most teams underestimate the first half because they only benchmark the final answer.

Why query rewriting exists in RAG

Retrieval augmented generation breaks when the user’s phrasing doesn’t match indexed content. A question like “Can I get my money back?” rarely shares tokens with a policy titled “Refund eligibility and process.” Rewriting maps the conversational query to a retrieval-friendly form: “refund policy eligibility timeline.”

The operation is almost always a second LLM call placed before the retrieval step. That call consumes tokens, opens a network connection, and serializes the request behind the primary generation. In a synchronous pipeline, the user sees the sum of both latencies.

Multi-turn conversations make this worse. A follow-up “What about international orders?” has no noun anchor. Rewriting must inject “refund policy for international orders” using prior context, which requires sending conversation history and increases prompt size and time-to-first-token.

How latency compounds in a naive pipeline

A typical RAG endpoint does three things in order: rewrite, retrieve, generate. If each step is sequential, the tail latency is additive.

def handle_request(user_query: str) -> str:
    rewritten = rewrite_query(user_query)   # 300ms
    docs = vector_search(rewritten)         # 50ms
    answer = generate_answer(rewritten, docs) # 1200ms
    return answer

The rewrite call is not free. Even a small model over a short prompt carries cold-start overhead on the provider side, TLS handshake, and inference queue time. If your generation model is a large frontier model, the rewrite might be 20% of total latency. If generation is a fast distilled model, the rewrite can be the dominant cost.

Sequential vs parallel rewrite

The mistake is assuming rewrite must block retrieval. In many designs, the original query can drive a fallback retrieval while the rewrite runs. You then fuse results.

import concurrent.futures

def handle_parallel(user_query: str) -> str:
    with concurrent.futures.ThreadPoolExecutor() as ex:
        f_rewrite = ex.submit(rewrite_query, user_query)
        f_orig_retrieval = ex.submit(vector_search, user_query)
        rewritten = f_rewrite.result()
        rewritten_retrieval = vector_search(rewritten)
        orig_docs = f_orig_retrieval.result()
    docs = rank_and_fuse([orig_docs, rewritten_retrieval])
    return generate_answer(rewritten, docs)

This cuts wall-clock time to max(rewrite, retrieve) + generate. But it doubles retrieval cost and complicates fusion. The query rewriting latency RAG teams actually experience is therefore a function of architecture, not just model speed.

Measuring the overhead: a minimal benchmark

You cannot optimize what you don’t measure. Build a harness that isolates the rewrite call from the rest.

import time, statistics
from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")

def timed_rewrite(q: str) -> tuple[str, float]:
    t0 = time.perf_counter()
    out = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role":"system","content":"Rewrite for retrieval."},
                  {"role":"user","content":q}],
        max_tokens=48)
    return out.choices[0].message.content, time.perf_counter() - t0

samples = []
for _ in range(200):
    _, dt = timed_rewrite("How do I cancel my subscription?")
    samples.append(dt * 1000)  # ms

print(f"p50={statistics.median(samples):.1f}ms p95={sorted(samples)[int(0.95*len(samples))]:.1f}ms")

If you route the rewrite through an OpenAI-compatible gateway such as n4n.ai, you get automatic fallback when a provider is degraded and per-token metering, which makes the cost of the extra call explicit. The p95 in this loop is what matters for user experience, not the average.

What the numbers mean

I’ve run variants of this against small models (e.g., 7B-class local or distilled cloud) and large ones. Small models routinely return in 80–150 ms on warm connections; frontier mini models sit at 200–400 ms; full frontier models exceed 600 ms. These are qualitative ranges from common public model behavior, not a published benchmark.

The key insight: query rewriting latency RAG overhead is dominated by time-to-first-token on the rewrite model. If you pick a model with low TTFT, the tax stays under 10% of a 2-second generation pipeline.

Model selection for the rewriter

Do not use your generation model for rewriting. The task is shallow: clarify intent, expand acronyms, add synonyms. A 70B model is overkill.

Options:

• Local 3–8B model with TensorRT or llama.cpp: 20–60 ms on commodity GPUs.
• Distilled cloud API (e.g., gpt-4o-mini, Claude Haiku): 150–300 ms.
• Heuristic rewriter (no LLM): regex + synonym map, <5 ms but brittle.

The tradeoff is quality. Heuristics fail on ambiguous context; small LLMs occasionally hallucinate a rewrite that hurts retrieval. Measure recall@5 on a labeled query set before committing.

{
  "rewrite_model": "gpt-4o-mini",
  "gen_model": "gpt-4o",
  "rewrite_timeout_ms": 350,
  "fallback_to_original_on_timeout": true
}

Batching rewrites across multiple user queries in a single API call amortizes connection overhead but increases p95 because the batch waits for the slowest item. For interactive RAG, batching the rewrite step is usually the wrong call.

Caching and invalidation

Rewrite outputs are deterministic for a given input string in most prompt setups. Cache them.

from functools import lru_cache

@lru_cache(maxsize=10_000)
def cached_rewrite(q: str) -> str:
    return rewrite_query(q)

In production, use a distributed cache keyed by query hash plus model version. Invalidation is trivial: bump the version when you change the rewrite prompt. This reduces query rewriting latency RAG pipelines to near zero for repeated or similar queries, which covers a large fraction of real traffic (greetings, FAQs, status checks). Avoid semantic caching for rewrites—exact match is safer because a slightly different rewrite can change retrieval results.

Tradeoffs: when to skip rewriting

Rewriting is not free, and not always needed.

• High-frequency short-tail queries: If 80% of queries are “password reset”, a static mapping beats an LLM call.
• Streaming-first UX: If you start generating before retrieval finishes, adding a blocking rewrite delays first token. Use parallel retrieval as above or drop rewrite.
• Cost-sensitive edge: On-device RAG on a phone cannot spare 300 ms for a cloud round trip. Use local heuristic.

Conversely, keep rewriting when:

• Queries are conversational with coreference (“it”, “that thing earlier”).
• Domain jargon mismatches user language (medical, legal).
• You already pay for a gateway call and can batch rewrite with other pre-processing.

Hybrid approach

A pragmatic middle ground: run a cheap heuristic rewrite always, then conditionally call an LLM rewrite only when heuristic confidence is low (e.g., retrieval score below threshold). This caps latency at the heuristic cost for most traffic while preserving quality where it matters.

Decisive takeaway

Measure the rewrite call in isolation, pick the smallest model that preserves retrieval recall, and run it concurrently with a baseline retrieval. If you do those three things, query rewriting latency RAG overhead stays under 15% of total request time and the retrieval quality lift justifies the complexity. If you cannot parallelize and are on a tight latency budget with simple queries, skip the LLM rewrite and use a cache-backed heuristic. The worst outcome is a sequential large-model rewrite you never benchmarked.