Designing agent loops for multi-step function calling

Designing agent loops multi-step function calling is where most LLM systems either become useful or collapse into flaky demos. The leap from a single chat completion to a reliable multi-turn agent that coordinates tools is a set of explicit engineering decisions about control flow, state, and failure boundaries. Treat the loop as a state machine, not a prompt.

1. Set a hard termination contract

An agent loop without a bounded step count will hang or burn tokens. Define max_iterations and an explicit task_complete tool the model must call to exit. Do not trust a natural-language “I’m done” string—models drift, and a regex match on output is a bug waiting to ship.

MAX_STEPS = 12

for step in range(MAX_STEPS):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=transcript,
        tools=TOOL_SCHEMAS,
        tool_choice="auto",
    )
    msg = resp.choices[0].message
    if msg.tool_calls and msg.tool_calls[0].function.name == "task_complete":
        break
    # dispatch tools...
else:
    raise TimeoutError("Agent exceeded step budget")

Add a secondary exit: a report_error tool for unrecoverable conditions. The loop should terminate cleanly and surface the error rather than spin. Pitfall: setting the budget too low truncates legitimate tasks; too high masks runaway loops. For typical retrieval-and-write workflows, 8–15 steps is a sane starting range. Revisit after observing real traces.

2. Model the tool surface as strict schemas

Designing agent loops multi-step function calling demands that every tool be a typed contract. Use JSON Schema with strict mode where the provider supports it. Avoid verbose descriptions that confuse the model; put nuances in parameter enums and clear names.

{
  "type": "function",
  "function": {
    "name": "query_orders",
    "description": "Fetch orders by status",
    "parameters": {
      "type": "object",
      "properties": {
        "status": {"type": "string", "enum": ["open", "shipped", "cancelled"]}
      },
      "required": ["status"]
    }
  }
}

Tradeoff: strict schemas reduce model creativity but eliminate a class of parsing errors. If a tool needs free-form input, accept a single string blob and parse it yourself. Version your tools—append _v2 when changing required parameters, because older transcripts may reference the old signature. Exposing more than ~20 tools in one turn degrades selection accuracy; split into domain-specific loops if needed.

3. Store state in a typed transcript

The model is stateless between calls. Your code owns the conversation. Append assistant messages (with tool_calls) and tool result messages with the correct tool_call_id. Use a typed wrapper to avoid role mismatches.

from pydantic import BaseModel

class ToolResult(BaseModel):
    tool_call_id: str
    content: dict

transcript.append(msg)  # assistant message with tool_calls
for call in msg.tool_calls:
    result = dispatch(call)
    transcript.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": json.dumps(result),
    })

Common pitfall: mutating prior tool results when retrying. Never rewrite history; append a new correction message instead. If a tool failed, return an error object as the tool content and let the model recover. Contaminating the transcript with patched successes hides bugs and breaks replay.

4. Choose execution shape: sequential vs parallel

A single model response can request multiple tool_calls when they are independent. Designing agent loops multi-step function calling that exploit this requires you to run them concurrently only when there are no data dependencies.

if len(msg.tool_calls) > 1:
    results = await asyncio.gather(*[run_tool(c) for c in msg.tool_calls])
else:
    results = [await run_tool(msg.tool_calls[0])]

The model does not always emit parallel calls even when it could. If you need guaranteed parallelism, force a plan step: ask the model to emit a list of independent operations first, then execute them. Tradeoff: extra round-trip latency versus simpler sequential code. For read-only fan-out (e.g., query three APIs), parallel wins. For write chains (create user → assign role), sequential is mandatory. Build a tiny dependency checker: if any call’s arguments reference another call’s output, serialize.

5. Make tools idempotent and side-effect aware

Tool execution must survive retries. A send_email tool called twice because of a network glitch should not spam the user. Use idempotency keys derived from tool_call_id.

async def send_email(call_id, to, body):
    if redis.get(f"done:{call_id}"):
        return {"status": "skipped"}
    # send...
    redis.set(f"done:{call_id}", 1)
    return {"status": "sent"}

Pitfall: treating tool calls as pure functions. They are not; the environment changes. Sandbox filesystem and network access. For destructive operations, insert a human-in-the-loop gate: return a pending_approval status and pause the loop until a callback arrives. The loop should treat approval as another tool message, not a special branch.

6. Route around provider degradation

Networks fail. Models get rate-limited. If you front your agent with an OpenAI-compatible gateway like n4n.ai, automatic fallback when a provider is rate-limited or degraded lets the loop continue without custom retry scaffolding. The client code stays identical because the endpoint honors the same schema.

from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key=KEY)
# same loop as above, fallback handled upstream

If you roll your own, implement exponential backoff with jitter and a circuit breaker per provider. That is a project in itself. Also forward provider cache-control hints where possible to avoid re-paying for stable system prompts on every step.

7. Meter and cap token drift

Each step adds context. Track cumulative usage from the response’s usage field. Stop or summarize when near a limit.

total_tokens = 0
for step in range(MAX_STEPS):
    resp = client.chat.completions.create(...)
    total_tokens += resp.usage.total_tokens
    if total_tokens > 60000:
        transcript = compress(transcript)

Tradeoff: compression loses detail. Use an explicit summarize_context tool call to shrink state rather than silent truncation. Per-token metering (as provided by your gateway) lets you attribute cost to a specific agent run and abort before a budget blows.

8. Test with replayable traces

Non-determinism makes debugging agent loops painful. Log the full transcript and tool I/O per step. Replay against a frozen model snapshot or a mock that returns recorded tool_calls.

Pitfall: writing unit tests that only check final answer. You must assert on intermediate tool selections to catch regressions in routing logic. Store traces as JSONL; write a harness that feeds them back through the loop with tools stubbed. This turns a probabilistic system into a testable pipeline.

Closing

Designing agent loops multi-step function calling is systems engineering with a probabilistic component. Bounded loops, strict schemas, explicit state, and provider resilience turn a prompt into a pipeline. Ship the loop, then optimize the model.