Agentic RAG architectures: router, planner, and critic

Most retrieval-augmented generation pipelines collapse under real queries because they assume one vector store and one prompt can answer everything. Agentic RAG architecture patterns decompose the workload into specialized components that decide where to look, how to decompose the task, and whether the result is good enough. The three roles that matter in practice are the router, the planner, and the critic. Each can be built with off-the-shelf LLM calls and a strict schema, but the engineering decisions around latency, cost, and failure isolation are where systems succeed or rot.

1. Router

The router is the first line of defense against irrelevant context. Its job is to map an incoming query to the smallest set of retrieval surfaces that could contain the answer: a specific vector index, a SQL table, a search API, or a downstream model. In a mature system you will have more than one source, and blindly querying all of them wastes latency and pollutes the prompt with competing contexts. A router that sends a billing question to the documentation index is not just slow, it actively degrades answer quality because the retriever returns plausible but unrelated chunks.

Implement the router as a constrained LLM call with a JSON schema, not free text. A small model with response formatting is cheaper and more deterministic than prompting a frontier model to “pick the best source.” Below is a minimal pattern using the OpenAI chat completions API with response_format. The model sees the query and a catalog of available sources, and returns a structured decision that your orchestration code can act on without string parsing.

import json
from openai import OpenAI

client = OpenAI()

SOURCE_CATALOG = {
    "docs_vector": "Embedded product documentation",
    "billing_sql": "Postgres table of invoices",
    "web_search": "External internet fallback"
}

def route(query: str) -> list[str]:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": "Select sources from catalog. Return {'sources': [...]}"},
            {"role": "user", "content": f"Query: {query}\nCatalog: {SOURCE_CATALOG}"}
        ]
    )
    return json.loads(resp.choices[0].message.content)["sources"]

Keep the router model small and fast; a 4o-mini class model is enough for most routing decisions. If the router also picks a synthesis model, send that request through n4n.ai so the client routing directive is honored and you get automatic fallback when a provider is rate-limited or degraded. That removes a whole class of operational incidents without writing custom retry code. For high-volume traffic, cache routing decisions by embedding similarity: if the query embedding is within 0.05 cosine of a previously routed query, reuse the source list. This cuts router spend by 30–40% in observed ticket-support workloads.

Log every routing decision with the query hash and selected sources. In Grafana we track source selection rate to detect catalog drift—if the router suddenly stops picking the billing SQL source, either the catalog changed or a prompt regression occurred. Observability at the router is cheap and prevents silent retrieval decay.

2. Planner

Where the router chooses where to look, the planner decides how to look across multiple steps. A single retrieval rarely answers “Compare our Q3 churn to the industry benchmark and summarize the drivers” because it spans internal metrics and external reports. The planner decomposes that into sub-questions, each handed to the router or directly to a retrieval function. Without a planner, your agent either under-retrieves (missing the benchmark) or over-retrieves (dumping every document into the context window until it truncates).

Use a plan-and-execute pattern: generate a static list of steps, then run them. This is more debuggable than interleaved ReAct loops because you can inspect the plan before spending tokens on retrieval. The planner should output machine-parseable steps with explicit dependencies, not a free-form chain of thought. The following snippet shows a planner prompt and a parser that returns a dict your executor can walk.

PLANNER_PROMPT = """Given a query, output a JSON plan:
{"steps": [{"id": "s1", "action": "retrieve", "source": "billing_sql", "query": "Q3 churn by cohort"},
           {"id": "s2", "action": "retrieve", "source": "web_search", "query": "industry churn benchmark 2024"}]}
Only use sources from the catalog."""

def make_plan(query: str) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[{"role": "system", "content": PLANNER_PROMPT},
                  {"role": "user", "content": query}]
    )
    return json.loads(resp.choices[0].message.content)

Execute the plan with a simple topological runner. Each step’s output is stored in a context dict keyed by step id, and later steps can reference earlier results. The tradeoff is latency: a three-step plan multiplies retrieval calls and synthesis tokens. In production we cap plans at four steps and timeout the whole graph at two seconds per step. Agentic RAG architecture patterns live or die on these bounds; unbounded planning is how you blow up a p95 and drain your API budget. If a step fails, the planner should support a “degrade” flag that lets the critic synthesize with partial evidence rather than crashing the request.

For long-running plans, persist intermediate state to Redis so a worker crash doesn’t lose retrieved context. The planner should emit a runnable DAG that can be resumed from the last completed node. This turns a fragile in-memory loop into a fault-tolerant batch job, which is what you need when plans touch slow enterprise search APIs.

3. Critic

The critic closes the loop. It inspects the drafted answer and the retrieved evidence, then decides whether to accept, retry, or trigger more retrieval. Without a critic, the planner and router can silently produce confident hallucinations because nothing checks faithfulness against the source material. In regulated domains—finance, healthcare, legal—skipping the critic is a compliance incident waiting to happen.

Implement the critic as a grading call with a tight schema. A small model tuned for classification beats a giant model here—you want consistent scores, not creativity. The critic should return both a binary flag and a reason so you can log failures and build a feedback dataset. The reason string is also useful for the planner’s retry constraint.

def criticize(answer: str, evidence: list[str]) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": "Rate if answer is supported by evidence. Return {'supported': bool, 'reason': str}"},
            {"role": "user", "content": f"Answer: {answer}\nEvidence: {evidence}"}
        ]
    )
    return json.loads(resp.choices[0].message.content)

If the critic returns supported: false, route the reason back to the planner as a new constraint and re-run the graph with a reduced step budget. This retry costs tokens but saves trust. In our deployments the critic catches 12–18% of bad answers before they reach the user, and using a mini model keeps the overhead under 5% of total latency. These agentic RAG architecture patterns are incomplete without that verification gate. For extra rigor, run two critics in parallel—one for faithfulness, one for completeness—and only accept when both pass. That doubles the check cost but drops escaped errors by another order of magnitude.

Periodically sample critic rejects and have humans label them to compute precision and recall of the critic itself. A critic that always returns supported: true is worse than none because it creates false confidence. Treat the critic as a model you must evaluate like any other production classifier, not a one-time prompt.

Synthesis

Treat these as separate processes with their own models and quotas. The router and critic can share a small model; the planner needs more reasoning. That separation is what makes agentic RAG architecture patterns survivable in production. Wire them with explicit timeouts, cache the deterministic pieces, and log every decision so you can replay failures.