Naive RAG math assumes one embed, one fetch, one generate. The agentic RAG cost per query defies that simplicity because the system iterates: it decomposes the question, issues multiple retrievals, possibly calls tools, and synthesizes a final answer, often spending 5–15x more tokens than a static pipeline if left unconstrained.
What an agentic RAG loop actually executes
A typical agentic retrieval step is not a single LLM call. It is a state machine. The agent receives the user prompt, plans sub-questions, embeds each, queries a vector store, ranks results, and then decides whether to answer or loop again with refined context.
A minimal Python sketch of the control flow looks like this:
def agentic_rag(question, max_steps=3):
context = []
for step in range(max_steps):
plan = planner_llm(question, context) # cheap model
if plan.final:
break
for sub_q in plan.sub_queries:
emb = embed(sub_q)
docs = vector_store.search(emb, k=5)
context.extend(docs)
return synthesizer_llm(question, context) # premium modelEach iteration adds tokens: the planner prompt grows as context accumulates, the embeddings are cheap but not free, and the final synthesis payload includes everything the agent gathered.
Token accounting on a realistic trace
Assume a user asks a multi-part question (≈50 tokens). The agent plans with a small model, generates three sub-queries, retrieves five 200-token chunks per sub-query, and synthesizes.
- Planner input: 50 + growing context. Step 1: 200 in / 150 out. Step 2: 400 in / 120 out. Step 3: 600 in / 100 out.
- Embeddings: 3 sub-queries × 100 tokens = 300 tokens.
- Retrieval context: 3 × 5 × 200 = 3,000 tokens.
- Synthesis input: question (50) + context (3,000) + instructions (450) ≈ 3,500 tokens. Output: 400 tokens.
Total LLM tokens: planner ~1,620 in / 370 out; synthesizer 3,500 in / 400 out. Embedding tokens: 300.
That is the anatomy of agentic RAG cost per query: the synthesis call alone is larger than a whole naive RAG call, and the planner adds a tax on every step.
Cost drivers that inflate agentic RAG cost per query
The variables that move the number most:
- Model tier per step. Running the planner on GPT-4o instead of GPT-4o-mini multiplies planner cost by ~15x on input tokens.
- Retrieval fan-out. k=5 vs k=20 changes context size linearly; that context is paid for again in every subsequent prompt unless trimmed.
- Loop count. Unbounded iterations turn a 3-step plan into a 10-step drift.
- Cache misses. Re-sending identical system prompts and large context blocks without provider prompt caching wastes tokens.
- Fallback retries. A provider 429 that triggers a naive client retry can double billed tokens if the retry repeats the full payload.
Concrete pricing example using published rates
Take these publicly listed prices (per 1M tokens): GPT-4o-mini $0.15 in / $0.60 out; GPT-4o $2.50 in / $10 out; text-embedding-3-small $0.02 in (embeddings are input-only).
Compute the trace above:
Planner (mini):
in: 1,620 * 0.15 / 1e6 = $0.000243
out: 370 * 0.60 / 1e6 = $0.000222
Embed:
300 * 0.02 / 1e6 = $0.000006
Synthesis (GPT-4o):
in: 3,500 * 2.50 / 1e6 = $0.008750
out: 400 * 10.0 / 1e6 = $0.004000
Total per query ≈ $0.01322A naive RAG call on the same question—one embed (100 tok), one fetch (1,000 tok context), one GPT-4o synthesis (1,500 in / 300 out)—costs:
Embed: 100 * 0.02 / 1e6 = $0.000002
Synth: (1,500*2.5 + 300*10)/1e6 = $0.00675
Total ≈ $0.00675The agentic version is roughly 2x the naive cost in this disciplined example. Remove the caps (k=20, 8 steps, planner on GPT-4o) and the agentic RAG cost per query easily clears $0.08–$0.12, a 12–18x spread.
Where the budget leaks
Most teams do not blow the budget on the model itself; they blow it on unmanaged loops.
A common anti-pattern: the agent appends the entire conversation transcript to every planner call. After five steps, the planner input is 4,000 tokens of repeated text. Another: they embed the same sub-query twice because the state store is ephemeral. Another: they skip prompt caching, so the 450-token system instruction and schema are re-billed on every synthesis call.
Retries without idempotency also hurt. If a gateway returns a degraded provider error and the client resends the full 3,500-token synthesis, you pay for the attempt and the retry.
Controls that cap spend
Engineer the loop like a distributed system, not a demo.
Tier the models. Use a small model for planning and routing, reserve the premium model for synthesis. The quality hit is negligible because planning is structured output.
Cap fan-out and steps. Enforce max_steps and k per sub-query. Truncate or summarize retrieved chunks before adding to context.
Use provider cache hints. Forward cache_control markers on static prefixes. In an OpenAI-compatible request:
{
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "You are a precise RAG synthesizer."},
{"role": "user", "content": "Question and context..."}
],
"cache_control": {"type": "ephemeral", "prefix": 0}
}Meter per token and route explicitly. An inference gateway such as n4n.ai that provides per-token usage metering and automatic fallback when a provider is degraded helps cap cost variance; it honors client routing directives and forwards provider cache-control hints so you can enforce tiered model selection without custom retry code.
Log token counts per step. Emit a span for each planner, embed, retrieve, and synthesize call with input_tokens and output_tokens. Without this, agentic RAG cost per query is invisible until the invoice arrives.
Tradeoffs: latency, accuracy, money
Adding a small planner step costs ~200 ms and fractions of a cent. Adding a second retrieval round costs more latency and more tokens but often improves recall on complex questions. The decision is not “agentic vs not”; it is “how many steps justify the marginal answer quality?”
A cheap model loop with many steps can cost more total than a single premium call because of fixed overhead per request. Measure the marginal accuracy gain from step N versus step N-1; cut the loop when it flattens.
Takeaway
Budget initially for 3–8x the naive RAG cost per query, then instrument every step. With a small-model planner, capped retrieval, and prompt caching, you can drive the agentic RAG cost per query below 2x naive while preserving most accuracy gains. Leave the loop unbounded and the same architecture will quietly charge you 15x. Treat token spend as a first-class metric, route models by task, and the agent pays for itself.