This gemini 3 function calling tutorial builds a multimodal agent that accepts an image and a text prompt, then decides whether to call a weather lookup or a vision descriptor tool. Gemini 3’s native function calling handles structured args; we wire the loop in Python with the official google-genai SDK and show exactly where the image bytes and tool responses meet.
Prerequisites
- Python 3.11 or newer (required for the
google-genaitype hints) google-genaiSDK (the maintained successor togoogle-generativeai)Pillowfor loading local images into bytes- A Gemini API key exported as
GEMINI_API_KEY - Comfort with Python dicts, basic HTTP, and synchronous control flow
If you have those installed, you can finish a working agent in about 20 minutes. No frontend, no vector DB, no framework—just the model and your code.
Project setup
Create a clean virtual environment and install the dependencies:
python -m venv .venv
source .venv/bin/activate
pip install google-genai pillowWe split the code into two modules so the tool implementations stay independent of the orchestration logic:
# tools.py
from typing import Any
def get_weather(lat: float, lon: float) -> dict[str, Any]:
# Stub: in production call Open-Meteo or similar
return {"temp_c": 14, "condition": "cloudy", "lat": lat, "lon": lon}
def describe_image_colors(image_path: str) -> dict[str, Any]:
from PIL import Image
img = Image.open(image_path).convert("RGB")
colors = img.getcolors(maxcolors=1_000_000)
top = sorted(colors, reverse=True)[:3]
return {"dominant": [{"count": c, "rgb": list(rgb)} for c, rgb in top]}# agent.py
import os
from google import genai
from google.genai import typesThat separation matters: Gemini only sees the schema, never your Python source.
Defining tools for Gemini 3
Gemini expects a FunctionDeclaration with a name, a description, and a JSON-Schema parameters object. The description is not decoration—the model uses it to choose between tools. We declare two:
# agent.py (continued)
weather_tool = types.FunctionDeclaration(
name="get_weather",
description="Get current weather for a latitude/longitude coordinate",
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"lat": types.Schema(type=types.Type.NUMBER, description="Decimal latitude"),
"lon": types.Schema(type=types.Type.NUMBER, description="Decimal longitude"),
},
required=["lat", "lon"],
),
)
image_tool = types.FunctionDeclaration(
name="describe_image_colors",
description="Return the three most dominant RGB colors of a local image file",
parameters=types.Schema(
type=types.Type.OBJECT,
properties={"image_path": types.Schema(type=types.Type.STRING)},
required=["image_path"],
),
)
tool_set = types.Tool(function_declarations=[weather_tool, image_tool])Keep parameter descriptions explicit. Gemini 3 will hesitate if lat could mean a string or a float.
Building the agent loop
The core is a generate_content call with tools=[tool_set]. If the model returns function_calls, we execute them locally and feed results back as function_response parts. The loop continues until the model returns plain text.
Sending multimodal input
Gemini 3 accepts inline image bytes alongside text in a single user turn. Load the file and wrap it in types.Part.from_bytes:
def build_multimodal_prompt(text: str, image_path: str) -> list[types.Content]:
with open(image_path, "rb") as f:
img_bytes = f.read()
mime = "image/png" if image_path.endswith(".png") else "image/jpeg"
return [
types.Content(
role="user",
parts=[
types.Part.from_text(text=text),
types.Part.from_bytes(data=img_bytes, mime_type=mime),
],
)
]For images already in cloud storage, use types.Part.from_uri(uri="gs://...", mime_type=...) instead—cheaper and avoids base64 overhead.
Handling function calls
We iterate over resp.function_calls. Each carries .name and .args. The response must be sent as a role="tool" content:
from tools import get_weather, describe_image_colors
def execute_tool(name: str, args: dict) -> dict:
if name == "get_weather":
return get_weather(**args)
if name == "describe_image_colors":
return describe_image_colors(**args)
raise ValueError(f"Unknown tool {name}")
def run_agent(text: str, image_path: str, max_steps: int = 5) -> str:
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
contents = build_multimodal_prompt(text, image_path)
for _ in range(max_steps):
resp = client.models.generate_content(
model="gemini-3-pro",
contents=contents,
tools=[tool_set],
)
if not resp.function_calls:
return resp.text or ""
for fc in resp.function_calls:
result = execute_tool(fc.name, fc.args)
contents.append(
types.Content(
role="tool",
parts=[
types.Part.from_function_response(
name=fc.name,
response={"result": result},
)
],
)
)
return "Agent exceeded step budget"The max_steps guard prevents a misconfigured schema from causing an infinite tool chain.
Running the agent end-to-end
With a local sample.png of a pale-blue sky, run:
if __name__ == "__main__":
answer = run_agent(
"What colors dominate this image, and is it warm outside at 37.77,-122.42?",
"sample.png",
)
print(answer)Expected output (wrapped for readability):
The image is dominated by light blue (rgb=[135,206,235], count=12033) and
white (rgb=[255,255,255], count=8021). At 37.77,-122.42 the current
temperature is 14°C with cloudy conditions, so it is not warm.Gemini 3 chained both tools: it called describe_image_colors on the uploaded bytes and get_weather with the coordinates you supplied, then synthesized a single answer. No orchestration framework required.
A second test with a cityscape and the prompt “Should I bring a jacket to 51.50,-0.12?” yields a similar pattern but only triggers get_weather, proving the model selects tools conditionally.
Debugging common failures
If you receive INVALID_ARGUMENT on the image part, the mime_type is almost certainly wrong. A .jpg file sent as image/png will be rejected.
If the model never calls your tool, tighten the description. “Fetch data” is ambiguous; “Get current weather for a latitude/longitude coordinate” is not. Gemini 3 also respects required fields strictly—if you omit them, it may hallucinate args.
When a tool raises, return the exception as a normal response dict rather than throwing out of execute_tool. The model can often self-correct if it sees {"error": "timeout"}.
Routing through a unified gateway
When you move past prototypes, provider outages become a real operational risk. An OpenAI-compatible gateway such as n4n.ai exposes Gemini 3 alongside 240+ other models behind one endpoint, and will automatically fall back to an alternative provider if Google’s API is rate-limited. Because it forwards the function-calling schema unchanged, the agent.py loop above only needs a different base_url and API key—no schema rewrites.
Extending the agent
Add more FunctionDeclaration objects to tool_set for database reads, calendar writes, or internal APIs. Gemini 3 supports parallel function calls: a single response may contain two function_calls that do not depend on each other. Our for fc in resp.function_calls: loop already executes them sequentially; for true parallelism you would ThreadPoolExecutor them before appending responses.
For production, wrap execute_tool in a timeout and a circuit breaker. Also persist contents to a session store so conversations survive restarts. The schema-forward design means you can swap the stub tools for real services without touching the model interaction code.
The rest of this gemini 3 function calling tutorial is about replacing stubs with authenticated APIs and adding guardrails—but the multimodal dispatch core is already production-shaped.