Building research agents forces a core architectural choice: parallel vs sequential search agents. The decision changes how you spend tokens, how fast you get answers, and how messy your context management becomes. Get it wrong and you either burn money on redundant fetches or stall on a single slow retrieval call.
Execution models
Sequential search
A sequential agent issues one search or tool call, ingests the result, then decides the next step. It is a tight loop where the model sees every prior result before choosing the next action:
async def sequential_research(query, max_steps=5):
ctx = [system_prompt(), user(query)]
for _ in range(max_steps):
resp = await llm.chat(ctx, model="gpt-4o-mini")
tool_call = extract_tool(resp)
if not tool_call:
return resp.text
result = await run_tool(tool_call) # blocking per step
ctx.append(tool_result(result))
return "max steps exceeded"This yields coherent trajectories but couples wall-clock time to step count. Each iteration re-sends the growing context, so you pay for accumulated tokens on every call unless your gateway forwards provider cache-control hints.
Parallel search
A parallel agent fans out multiple independent searches at once, usually after a planner generates sub-queries, then merges results:
async def parallel_research(query):
subqueries = await planner.split(query) # e.g., 4 sub-questions
tasks = [run_search(sq) for sq in subqueries]
results = await asyncio.gather(*tasks, return_exceptions=True)
clean = [r for r in results if not isinstance(r, Exception)]
merged = await synthesize(clean)
return await llm.chat([user(merged)])Fan-out cuts latency when sub-tasks are independent, but you lose the cumulative context that sequential search naturally builds. You also inherit race conditions and partial failure handling.
Dimensions compared
Capabilities
Sequential search excels at dependent chains: find a paper, then search its citations, then check the author’s later work. Each step informs the next. Parallel search excels at breadth: market landscape scans, competitor feature matrices, or “find all mentions of X across N sources.”
Parallel agents can simulate sequential dependency by running a pre-plan, but they routinely fetch irrelevant documents because they cannot react to intermediate findings. Sequential agents handle evolving hypotheses natively—a legal researcher following a citation trail is a sequential job; a procurement agent comparing 12 vendor specs is parallel.
Price/cost model
Both pay per token for LLM calls and per query for retrieval. Sequential search often uses smaller models per step because each step is narrow; parallel search typically calls a planner (larger model) once, then many small retrievals.
The hidden cost in parallel is redundant context: you may pull 10 documents when 3 would suffice after a first sequential filter. Sequential spends more on round-trip LLM calls but less on wasted retrieval. With per-token usage metering, sequential usually shows lower variance in spend. If you replay the same system prompt across steps, a gateway that honors cache-control can drop repeat input token cost dramatically—a bigger win for sequential loops.
Latency/throughput
Sequential latency is the sum of step times: T = Σ(t_llm + t_tool). Parallel latency is max(t_planner, max(t_tool)) + t_synth. For a 4-step chain where each LLM call is ~500ms and each search ~300ms, sequential totals ~3.2s; parallel with a 600ms planner and 300ms searches totals ~1.1s plus synthesis. That 3x is real but assumes true independence.
Under provider degradation, sequential fails gracefully per step; parallel suffers partial gaps unless you implement fallback. An inference gateway like n4n.ai can mask provider outages with automatic fallback, which makes parallel fan-out less risky when a single model endpoint hiccups.
Ergonics
Sequential code is easy to debug: print the loop, see each decision. Parallel requires handling coroutine races, result ordering, and partial failures. Most agent frameworks (LangGraph, LlamaIndex) give sequential a first-class AgentExecutor loop; parallel needs custom asyncio or map-reduce nodes.
Context window management is simpler sequentially: you append and trim. Parallel forces you to pack multiple results into one prompt without exceeding limits, often requiring pre-summarization per branch. Writing unit tests for sequential is straightforward; for parallel you must mock asyncio.gather scenarios with mixed successes.
Ecosystem
Sequential patterns dominate existing agent examples because they map to ReAct. Parallel is newer: seen in “deep research” products that spawn worker agents. Libraries like dspy or tanuki support both, but parallel often means rolling your own orchestration.
Model routing matters: if you use an OpenAI-compatible endpoint that addresses 240+ models, you can send planner calls to a reasoning model and retrieval synthesis to a cheap one. That flexibility applies to both, but parallel benefits more because it issues more heterogeneous calls in a single user request.
Limits
Sequential hits a hard ceiling on depth vs time: users abandon after 30s of silence. Parallel hits context explosion: merging 8 search results of 4k tokens each exceeds many model windows unless you compress.
Both share a failure mode: poorly scoped queries produce garbage regardless of topology. Neither fixes bad retrieval. Rate limits also bite differently—sequential spreads load over time; parallel spikes concurrent requests and can trip provider quotas.
Comparison table
| Dimension | Sequential search | Parallel search |
|---|---|---|
| Dependency handling | Native, step-by-step | Requires pre-planning |
| Typical latency | Sum of steps (high) | Max of branches (low) |
| Token cost shape | More LLM calls, less waste | Fewer LLM calls, more retrieval waste |
| Debugging | Trivial loop inspection | Race conditions, partial results |
| Best for | Iterative, dependent chains | Independent breadth scans |
| Context risk | Unbounded but trimable | Merged explosion without summarization |
| Failure mode | Slow stall on bad step | Silent partial gaps |
| Rate limit impact | Spread over time | Concurrent spike |
Which to choose
Use sequential search agents when
- The task is exploratory with unknown branching (e.g., “trace the lineage of this bug fix across repos”).
- You need auditability: every hop logged for compliance or post-mortem.
- Latency budget is >10s and cost predictability matters more than speed.
- You run on a single model endpoint without fallback; sequential contains blast radius to one step.
Use parallel search agents when
- You have a clear decomposition: “compare AWS, GCP, Azure SSO docs” splits cleanly into independent branches.
- User expects sub-5s response and you can afford one planner call plus synthesis.
- Your gateway supports fallback so a single failed branch doesn’t kill the run.
- You can pre-summarize each branch to control context size before merge.
Hybrid is the default in production
Start sequential. It is easier to ship and reveals the natural decomposition of your domain. Once you see repeated independent sub-queries, extract them into a parallel fan-out behind a feature flag. Most production research agents end up as a sequential planner that emits a bounded parallel search batch, then sequentially refines the merged evidence.
The parallel vs sequential search agents decision should be economic, not ideological. If you are building on an OpenAI-compatible endpoint that addresses 240+ models, route the planner to a reasoning model and the synthesizer to a compact one, and let the orchestration layer decide fan-out based on query structure rather than a fixed pattern.