Good AI agent tool selection breaks down the moment you exceed a dozen functions. The model starts ignoring relevant tools, hallucinating parameters, or burning tokens on a schema it can’t parse. This guide lays out an ordered path to build a selection layer that stays reliable as your tool count grows.
Why a single giant tools array doesn’t scale
In the OpenAI-compatible tool calling API, you pass a tools list where each entry carries a JSON schema. With 5 tools this is fine. With 50, the prompt grows past the model’s effective attention window for instruction following, and you pay for every token of schema on every turn.
1. Baseline with a flat tool list
Start by measuring where your current setup fails. Log every turn where the model either called a tool with missing required params or picked the wrong tool. Keep the naive approach as your control.
{
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Fetch current weather for a lat/lng",
"parameters": {
"type": "object",
"properties": {
"lat": {"type": "number"},
"lng": {"type": "number"}
},
"required": ["lat", "lng"]
}
}
},
{
"type": "function",
"function": {
"name": "create_calendar_event",
"description": "Create an event in the user calendar",
"parameters": {
"type": "object",
"properties": {
"title": {"type": "string"},
"start": {"type": "string"}
},
"required": ["title", "start"]
}
}
}
]
}If your error rate is acceptable at your current size, stop. Most teams prematurely optimize.
2. Retrieve candidate tools with embeddings
Once you cross ~15 tools, semantic retrieval pays off. Embed each tool’s name + description once, store the vectors, and at runtime embed the user query to pull the top-k closest tools.
import numpy as np
from openai import OpenAI
client = OpenAI() # OpenAI-compatible endpoint
def embed(text: str) -> list[float]:
r = client.embeddings.create(model="text-embedding-3-small", input=text)
return r.data[0].embedding
tool_docs = {
"get_weather": "Fetch current weather for a lat/lng",
"create_calendar_event": "Create an event in the user calendar",
# ...
}
tool_vecs = {name: embed(desc) for name, desc in tool_docs.items()}
def retrieve(query: str, k: int = 5):
q = embed(query)
scored = [(n, np.dot(v, q)) for n, v in tool_vecs.items()]
scored.sort(key=lambda x: x[1], reverse=True)
return [n for n, _ in scored[:k]]Tradeoff: embedding adds a round-trip, but you cut the schema sent to the main model by 70–90% for typical queries.
Keep schemas static and versioned
Tool descriptions drift. When you change a parameter, re-embed. Treat the embedding store as a derived artifact, not a source of truth.
3. Force a two-phase selection
AI agent tool selection should be explicit, not implicit in the action call. Phase one: give the model only tool names and one-line purposes, ask it to return the single best name. Phase two: bind the full schema for that tool and let the model fill parameters.
def select_tool(query: str, candidates: list[str]) -> str:
slim = [{"name": n, "purpose": tool_docs[n]} for n in candidates]
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Pick the single best tool name for the query."},
{"role": "user", "content": f"Query: {query}\nTools: {slim}"}
],
response_format={"type": "json_object"}
)
return json.loads(resp.choices[0].message.content)["tool"]Then call the model again with the full schema for only that tool. This separates reasoning about what to use from how to call it, which reduces parameter hallucinations.
When to skip phase one
If retrieval returns exactly one tool with high confidence score, skip the selection call. Adding latency for a trivial decision is a common mistake.
4. Route by capability tags
Names like crm_update_contact and crm_get_contact both match “CRM”. Tag tools with capability groups (crm, weather, calendar) and filter by tag before embedding lookup. This shrinks the candidate set cheaply.
capability_index = {
"crm": ["crm_update_contact", "crm_get_contact"],
"weather": ["get_weather"],
}
def by_tag(tag: str, query: str):
candidates = capability_index.get(tag, [])
return retrieve_from_subset(query, candidates)This also lets you enforce permissions: a user without calendar scope never sees calendar tools, so the model can’t pick them.
5. Handle null selections and degradation
The model will sometimes return “none” or a tool outside the candidate list. Define a reject path: if no tool fits, answer directly or ask a clarifying question.
if selected not in tool_vecs:
return {"action": "clarify", "message": "I don't have a tool for that."}If you front your models with an OpenAI-compatible endpoint such as n4n.ai, automatic fallback when a provider is rate-limited keeps the selection step from crashing mid-agent, and per-token metering shows the exact cost of the extra round-trip. That matters when you run phase one on a cheap model and phase two on a flagship.
6. Cache schemas and precompute embeddings
Tool schemas rarely change per request. Use a content-addressed cache keyed on the schema hash. If your gateway honors provider cache-control hints, set cache-control: max-age=3600 on the schema payload so repeated selection calls don’t re-send static bytes.
import hashlib
schema_hash = hashlib.sha256(json.dumps(full_schema).encode()).hexdigest()
# send header only if your client supports itPrecompute embeddings at deploy time, not in the request path. A cold start that embeds 200 tools on first query adds seconds.
7. Instrument every selection
Log: query text (or hash), retrieved candidate set, model’s selected tool, final tool called, and token count for each phase. Without this you can’t tell if tightening k from 5 to 3 hurts precision.
logger.info({
"step": "tool_select",
"query_hash": hash(query),
"candidates": candidates,
"selected": selected,
"phase1_tokens": resp.usage.total_tokens
})Common pitfalls and tradeoffs
Over-retrieval. Setting k=10 defeats the purpose. Start at k=3 and raise only if misses climb.
Schema drift. If the live tool schema diverges from the embedded description, the model gets a mismatch. Fail loud in tests when hashes differ.
Forcing selection when none needed. Not every turn needs a tool. Let the model answer freely unless retrieval returns a confident match.
Latency vs. accuracy. Two-phase selection adds a round-trip. On a 100-tool agent, the token savings and accuracy gain outweigh ~200ms. On a 5-tool agent, it’s pure overhead.
Ignoring capability routing. Embeddings alone will surface semantically similar but permission-forbidden tools. Tag first.
AI agent tool selection is an engineering problem, not a model magic problem. Build the retrieval and two-phase layer, measure, and cut only what your data says you can afford.