Semantic Kernel agent tutorial: GPT-4o and Claude fallback

Semantic Kernel’s planner architecture makes it straightforward to build agents that reason over tools, but production workloads need model redundancy. This tutorial shows how to wire a Semantic Kernel agent against a single OpenAI-compatible endpoint that serves GPT-4o and automatically fails over to Claude when the primary provider is rate-limited or degraded. You’ll configure the planner, inject routing directives, and verify fallback behavior without changing your application code.

Prerequisites

• Python 3.10+
• An n4n.ai API key (or any OpenAI-compatible endpoint that supports multiple models and automatic fallback)
• semantic-kernel >= 1.0.0, openai >= 1.0.0, python-dotenv

pip install semantic-kernel openai python-dotenv

Create a .env file:

N4N_API_KEY=sk-your-key
N4N_BASE_URL=https://api.n4n.ai/v1

The base URL is the only configuration change required. The endpoint honors model routing directives and forwards provider cache-control hints, so your Semantic Kernel code stays standard.

Project structure

sk-agent-fallback/
├── .env
├── main.py
├── plugins/
│   ├── __init__.py
│   └── math_plugin.py
└── requirements.txt

Define a plugin for the planner

Semantic Kernel planners operate over registered functions. We’ll expose a tiny math plugin so the agent has something to reason about.

# plugins/math_plugin.py
from semantic_kernel.functions import kernel_function

class MathPlugin:
    @kernel_function(
        name="add",
        description="Add two numbers"
    )
    def add(self, a: float, b: float) -> float:
        return a + b

    @kernel_function(
        name="multiply",
        description="Multiply two numbers"
    )
    def multiply(self, a: float, b: float) -> float:
        return a * b

    @kernel_function(
        name="divide",
        description="Divide a by b"
    )
    def divide(self, a: float, b: float) -> float:
        if b == 0:
            raise ValueError("Cannot divide by zero")
        return a / b

# plugins/__init__.py
from .math_plugin import MathPlugin

__all__ = ["MathPlugin"]

Configure the kernel with the unified endpoint

The kernel uses the standard OpenAI client pointed at the n4n.ai base URL. No custom HTTP logic required.

# main.py
import os
import asyncio
from dotenv import load_dotenv
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion
from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
from semantic_kernel.contents import ChatHistory
from semantic_kernel.functions import KernelArguments
from plugins.math_plugin import MathPlugin

load_dotenv()

def build_kernel() -> Kernel:
    kernel = Kernel()
    
    # Primary model directive — GPT-4o
    # The endpoint will automatically fall back to Claude on 429/5xx
    chat_service = OpenAIChatCompletion(
        ai_model_id="gpt-4o",
        api_key=os.getenv("N4N_API_KEY"),
        endpoint=os.getenv("N4N_BASE_URL"),
        service_id="primary"
    )
    kernel.add_service(chat_service)
    
    # Register plugins
    kernel.add_plugin(MathPlugin(), plugin_name="math")
    
    return kernel

Build the planning agent

We’ll use the FunctionCallingStepwisePlanner (available in SK 1.0+) which lets the model decide when to call tools and when to respond.

# main.py (continued)
from semantic_kernel.planners import FunctionCallingStepwisePlanner
from semantic_kernel.planners.function_calling_stepwise_planner import (
    FunctionCallingStepwisePlannerOptions
)

async def run_agent(kernel: Kernel, goal: str) -> str:
    planner = FunctionCallingStepwisePlanner(
        service_id="primary",
        options=FunctionCallingStepwisePlannerOptions(
            max_iterations=10,
            max_tokens=4000
        )
    )
    
    # The planner builds its own chat history internally
    result = await planner.invoke(kernel, goal)
    return str(result)

Test the primary path

Run a goal that requires multiple tool calls. This exercises the planner’s reasoning loop against GPT-4o.

# main.py (continued)
async def main():
    kernel = build_kernel()
    
    goal = (
        "Calculate (15 * 4) + (100 / 5) - 7. "
        "Use the math plugin for each operation and show your work."
    )
    
    print(f"Goal: {goal}\n")
    print("Running planner against GPT-4o...\n")
    
    result = await run_agent(kernel, goal)
    print(f"Result:\n{result}")

if __name__ == "__main__":
    asyncio.run(main())

Expected output (formatting may vary):

Goal: Calculate (15 * 4) + (100 / 5) - 7. Use the math plugin for each operation and show your work.

Running planner against GPT-4o...

Step 1: Calling math.multiply with a=15, b=4
Step 2: Calling math.divide with a=100, b=5
Step 3: Calling math.add with a=60, b=20
Step 4: Calling math.add with a=80, b=-7

Result: 73

The planner correctly decomposed the expression and chained three tool calls. Check your n4n.ai dashboard — the request shows model: gpt-4o and provider: openai.

Force a fallback to verify behavior

You can simulate provider degradation by sending a routing directive that targets an overloaded model, or by temporarily setting an invalid key for the primary provider. The cleaner approach: use the x-n4n-routing header via the extra_headers parameter to pin a model that will trigger fallback.

# main.py — add this import at top
from openai import AsyncOpenAI

async def test_fallback(kernel: Kernel):
    """
    Demonstrates fallback by forcing a request to a model
    that the gateway will route away from (e.g., a deprecated alias).
    The gateway returns Claude with a `x-n4n-provider: anthropic` header.
    """
    # Access the underlying client to inject routing hint
    chat_service = kernel.get_service("primary")
    client: AsyncOpenAI = chat_service.client  # type: ignore
    
    # Create a one-off completion with routing directive
    # "claude-3-opus" at n4n.ai maps to Claude 3 Opus; if capacity is full,
    # the gateway falls back to Sonnet or Haiku automatically.
    response = await client.chat.completions.create(
        model="claude-3-opus-20240229",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "What is 2 + 2?"}
        ],
        extra_headers={
            "x-n4n-routing": "prefer:claude,fallback:auto"
        },
        max_tokens=50
    )
    
    print(f"Fallback test response: {response.choices[0].message.content}")
    print(f"Provider header: {response.headers.get('x-n4n-provider')}")
    print(f"Model used: {response.model}")

Add a call to test_fallback(kernel) at the end of main() and run again. You’ll see:

Fallback test response: 2 + 2 = 4
Provider header: anthropic
Model used: claude-3-5-sonnet-20241022

The gateway honored the preference for Claude, detected capacity pressure on Opus, and transparently served Sonnet instead. Your Semantic Kernel code never saw the switch.

Wire fallback into the planner loop

For production, you want the planner itself to survive provider failures mid-execution. The gateway handles this at the HTTP layer — each planner step is a separate chat.completions call, and each call independently falls back. But you can also instruct the gateway to prefer a provider order for the entire session.

# main.py — update build_kernel()
def build_kernel() -> Kernel:
    kernel = Kernel()
    
    chat_service = OpenAIChatCompletion(
        ai_model_id="gpt-4o",
        api_key=os.getenv("N4N_API_KEY"),
        endpoint=os.getenv("N4N_BASE_URL"),
        service_id="primary",
        # These become default headers on every request from this service
        default_headers={
            "x-n4n-routing": "prefer:openai,fallback:anthropic"
        }
    )
    kernel.add_service(chat_service)
    kernel.add_plugin(MathPlugin(), plugin_name="math")
    
    return kernel

The x-n4n-routing header accepts a comma-separated preference list. Values: openai, anthropic, google, auto. The gateway tries each in order until one succeeds. This is the single line that gives your Semantic Kernel agent multi-provider resilience.

Handle streaming and token metering

If you stream planner steps, the gateway returns standard SSE chunks. Token usage accumulates per request and appears in the final chunk’s usage field and the x-n4n-usage response header (prompt_tokens, completion_tokens, total_tokens).

async def run_agent_streaming(kernel: Kernel, goal: str):
    planner = FunctionCallingStepwisePlanner(
        service_id="primary",
        options=FunctionCallingStepwisePlannerOptions(
            max_iterations=10,
            max_tokens=4000
        )
    )
    
    # The planner doesn't expose streaming directly in 1.0,
    # but you can stream the underlying chat completion service:
    chat_service = kernel.get_service("primary")
    
    history = ChatHistory()
    history.add_system_message(
        "You are a planner. Use the math plugin to solve the user's goal."
    )
    history.add_user_message(goal)
    
    # Manual planning loop with streaming (simplified)
    arguments = KernelArguments()
    async for chunk in chat_service.get_streaming_chat_message_contents(
        chat_history=history,
        settings=chat_service.get_prompt_execution_settings_class()(
            function_choice_behavior=FunctionChoiceBehavior.Auto()
        ),
        kernel=kernel,
        arguments=arguments
    ):
        print(chunk.content, end="", flush=True)
    print()

Observability: correlate planner steps with provider

Each planner iteration generates a distinct request ID. Log the x-n4n-request-id and x-n4n-provider headers alongside your application logs. When debugging a failed step, you can tell exactly which provider served it and whether a fallback occurred.

# Add to your logging middleware or custom HTTP client wrapper
import logging

class ProviderLoggingHandler(logging.Handler):
    def emit(self, record):
        if hasattr(record, 'n4n_provider'):
            print(f"[SK Step] provider={record.n4n_provider} "
                  f"request_id={record.n4n_request_id} "
                  f"model={record.n4n_model}")

Common pitfalls

Planner loops infinitely — Set max_iterations in FunctionCallingStepwisePlannerOptions. The default is 15; tighten it for latency-sensitive paths.

Function schema mismatch — Ensure every @kernel_function has a proper description and type annotations. The planner builds its function-calling schema from these.

Fallback changes model behavior — GPT-4o and Claude have different function-calling styles. Test your prompts against both. If the planner produces malformed calls under Claude, add a few-shot example to the system prompt showing the expected tool_calls format.

Rate limits mid-session — The gateway retries with exponential backoff before falling back. If you see 429s in your logs, the fallback already happened. No client-side retry logic needed.

What you’ve built

• A Semantic Kernel agent that plans and executes multi-step tool use
• Single-endpoint configuration with declarative provider preference
• Automatic fallback from GPT-4o to Claude (or any provider the gateway supports) on rate limits or errors
• Per-request token metering and provider attribution via response headers
• Zero application-code changes when the gateway adds new models

The planner, plugins, and kernel setup are all standard Semantic Kernel. The only n4n.ai-specific lines are the base URL and the x-n4n-routing header. Swap the endpoint, and the same code runs against any OpenAI-compatible gateway that honors that header.