n4nAI
Open console →
API integrationTutorialMulti-Provider Fallback Code Patterns

Multi-provider LLM fallback code patterns in Go

A hands-on Go tutorial for building multi provider llm fallback golang patterns: chain providers, handle rate limits, and degrade gracefully.

n4n Team2 min read440 words

Audio narration

Coming soon — every post will get a voice note here.

Building resilient LLM integrations means planning for provider outages, rate limits, and silent degradations. This tutorial walks through a concrete multi provider llm fallback golang implementation that chains OpenAI, Anthropic, and a local Ollama model, degrading gracefully when each fails.

Prerequisites

  • Go 1.21 or newer
  • OpenAI API key (OPENAI_API_KEY)
  • Anthropic API key (ANTHROPIC_API_KEY)
  • A running Ollama instance on localhost:11434 (optional, but the code compiles without it)
  • Basic comfort with net/http and context

Initialize the module:

go mod init fallback

We will use only the standard library. No external SDKs, so the fallback logic stays transparent.

Defining a common provider interface

A fallback chain is only as clean as its abstraction. Define a minimal contract that every provider satisfies:

package main

import "context"

type CompletionRequest struct {
	Prompt    string
	MaxTokens int
}

type CompletionResponse struct {
	Text  string
	Model string
}

type Provider interface {
	Name() string
	Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error)
}

This interface lets us treat OpenAI, Anthropic, and a local model identically. The Complete method must respect context cancellation and return a distinguishable error on rate limiting.

Implementing the OpenAI provider

We target the chat completions endpoint. The code below sends a minimal request and maps errors:

type OpenAIProvider struct {
	APIKey string
	Model  string
}

func (p OpenAIProvider) Name() string { return "openai/" + p.Model }

func (p OpenAIProvider) Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error) {
	body := map[string]any{
		"model":       p.Model,
		"max_tokens":  req.MaxTokens,
		"messages":    []map[string]string{{"role": "user", "content": req.Prompt}},
	}
	b, _ := json.Marshal(body)
	httpReq, _ := http.NewRequestWithContext(ctx, "POST", "https://api.openai.com/v1/chat/completions", bytes.NewReader(b))
	httpReq.Header.Set("Authorization", "Bearer "+p.APIKey)
	httpReq.Header.Set("Content-Type", "application/json")

	resp, err := http.DefaultClient.Do(httpReq)
	if err != nil {
		return CompletionResponse{}, err
	}
	defer resp.Body.Close()

	if resp.StatusCode == 429 {
		return CompletionResponse{}, &RateLimitError{Provider: p.Name()}
	}
	if resp.StatusCode != 200 {
		return CompletionResponse{}, fmt.Errorf("openai status %d", resp.StatusCode)
	}

	var out struct {
		Choices []struct {
			Message struct {
				Content string `json:"content"`
			} `json:"message"`
		} `json:"choices"`
	}
	json.NewDecoder(resp.Body).Decode(&out)
	return CompletionResponse{Text: out.Choices[0].Message.Content, Model: p.Name()}, nil
}

We define a sentinel error type for rate limits so the chain can log differently:

type RateLimitError struct{ Provider string }

func (e *RateLimitError) Error() string { return e.Provider + " rate limited" }

Implementing the Anthropic provider

Anthropic’s messages API uses a different shape and requires a version header:

type AnthropicProvider struct {
	APIKey string
	Model  string
}

func (p AnthropicProvider) Name() string { return "anthropic/" + p.Model }

func (p AnthropicProvider) Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error) {
	body := map[string]any{
		"model":      p.Model,
		"max_tokens": req.MaxTokens,
		"messages":   []map[string]string{{"role": "user", "content": req.Prompt}},
	}
	b, _ := json.Marshal(body)
	httpReq, _ := http.NewRequestWithContext(ctx, "POST", "https://api.anthropic.com/v1/messages", bytes.NewReader(b))
	httpReq.Header.Set("x-api-key", p.APIKey)
	httpReq.Header.Set("anthropic-version", "2023-06-01")
	httpReq.Header.Set("Content-Type", "application/json")

	resp, err := http.DefaultClient.Do(httpReq)
	if err != nil {
		return CompletionResponse{}, err
	}
	defer resp.Body.Close()

	if resp.StatusCode == 429 {
		return CompletionResponse{}, &RateLimitError{Provider: p.Name()}
	}
	if resp.StatusCode != 200 {
		return CompletionResponse{}, fmt.Errorf("anthropic status %d", resp.StatusCode)
	}

	var out struct {
		Content []struct {
			Text string `json:"text"`
		} `json:"content"`
	}
	json.NewDecoder(resp.Body).Decode(&out)
	return CompletionResponse{Text: out.Content[0].Text, Model: p.Name()}, nil
}

Adding a local Ollama provider

For air-gapped or cheap fallback, Ollama speaks a simple JSON API:

type OllamaProvider struct {
	Model string
}

func (p OllamaProvider) Name() string { return "ollama/" + p.Model }

func (p OllamaProvider) Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error) {
	body := map[string]any{
		"model":  p.Model,
		"prompt": req.Prompt,
		"options": map[string]any{
			"num_predict": req.MaxTokens,
		},
	}
	b, _ := json.Marshal(body)
	httpReq, _ := http.NewRequestWithContext(ctx, "POST", "http://localhost:11434/api/generate", bytes.NewReader(b))
	httpReq.Header.Set("Content-Type", "application/json")

	resp, err := http.DefaultClient.Do(httpReq)
	if err != nil {
		return CompletionResponse{}, err
	}
	defer resp.Body.Close()

	if resp.StatusCode != 200 {
		return CompletionResponse{}, fmt.Errorf("ollama status %d", resp.StatusCode)
	}

	var out struct {
		Response string `json:"response"`
	}
	json.NewDecoder(resp.Body).Decode(&out)
	return CompletionResponse{Text: out.Response, Model: p.Name()}, nil
}

Building the fallback chain

The core pattern: iterate providers, enforce a per-call timeout, and skip to the next on any error. Distinguish rate limits for observability.

type FallbackChain struct {
	providers []Provider
}

func (c FallbackChain) Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error) {
	var lastErr error
	for _, p := range c.providers {
		callCtx, cancel := context.WithTimeout(ctx, 8*time.Second)
		resp, err := p.Complete(callCtx, req)
		cancel()

		if err == nil {
			return resp, nil
		}

		if _, ok := err.(*RateLimitError); ok {
			log.Printf("warn: %s rate limited, trying next", p.Name())
		} else {
			log.Printf("warn: %s failed: %v", p.Name(), err)
		}
		lastErr = err
	}
	return CompletionResponse{}, fmt.Errorf("all providers exhausted: %w", lastErr)
}

Order matters. Put the highest-quality provider first, the cheapest local model last.

Wiring it together

func main() {
	chain := FallbackChain{providers: []Provider{
		OpenAIProvider{APIKey: os.Getenv("OPENAI_API_KEY"), Model: "gpt-4o-mini"},
		AnthropicProvider{APIKey: os.Getenv("ANTHROPIC_API_KEY"), Model: "claude-3-haiku-20240307"},
		OllamaProvider{Model: "llama3"},
	}}

	req := CompletionRequest{Prompt: "Explain fallback in one sentence.", MaxTokens: 64}
	resp, err := chain.Complete(context.Background(), req)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("answered by %s: %s\n", resp.Model, resp.Text)
}

If you would rather not hand-roll this, a gateway like n4n.ai provides automatic fallback across 240+ models behind one OpenAI-compatible endpoint, but the pattern above is exactly what runs inside such a system.

Expected output

With all providers healthy, you should see a single answer and no warnings:

answered by openai/gpt-4o-mini: Fallback routes requests to backup systems when primary ones fail.

If OpenAI returns 429, the log shows:

warn: openai/gpt-4o-mini rate limited, trying next
answered by anthropic/claude-3-haiku-20240307: Fallback delegates to another provider on failure.

If both cloud providers are down, the local Ollama responds:

warn: openai/gpt-4o-mini failed: openai status 503
warn: anthropic/claude-3-haiku-20240307 failed: anthropic status 503
answered by ollama/llama3: Fallback uses a local model when clouds are unreachable.

Handling partial failures and retries

The naive chain tries each provider once. In production, add a bounded retry with jitter for transient 5xx errors before falling back. Wrap the call in a small retry helper:

func retry(ctx context.Context, p Provider, req CompletionRequest, attempts int) (CompletionResponse, error) {
	var err error
	for i := 0; i < attempts; i++ {
		var resp CompletionResponse
		resp, err = p.Complete(ctx, req)
		if err == nil {
			return resp, nil
		}
		if _, ok := err.(*RateLimitError); ok {
			return CompletionResponse{}, err // don't retry rate limits
		}
		select {
		case <-ctx.Done():
			return CompletionResponse{}, ctx.Err()
		case <-time.After(time.Duration(i+1) * 100 * time.Millisecond):
		}
	}
	return CompletionResponse{}, err
}

Swap p.Complete with retry inside the loop. This reduces false fallbacks on blips.

Closing notes

The multi provider llm fallback golang pattern is fundamentally a loop with timeouts and error classification. Keep the interface tiny, log which provider answered, and put the local model last. When you outgrow self-managed fallback, a gateway that honors client routing directives and forwards cache-control hints can centralize the logic—but the code above is the bedrock.

Make sure to set GOMAXPROCS appropriately and never reuse http.DefaultClient in high-throughput services; build a client with a timeout. That’s the whole mechanism.

Tagsfallbackgolangmulti-providercode-pattern

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationMulti-provider LLM fallback: a Python code patternA hands-on Python tutorial for building a multi provider llm fallback python pattern with OpenAI and Anthropic SDKs, plus graceful degradation.API integrationMulti-provider LLM fallback code patterns for Node.jsBuild a multi provider llm fallback nodejs tutorial: sequential and parallel patterns, timeouts, circuit breakers, and cache hints.API integrationA fallback pattern for LLM APIs using try/except chainsBuild a Python try except fallback pattern llm api tutorial: chain multiple providers with graceful degradation, retries, and clear error handling.API integrationCircuit breakers for multi-provider LLM fallback codeA practical guide to implementing a circuit breaker llm fallback pattern across multiple LLM providers, including Python code, tradeoffs, and common pitfalls.API integrationHandling HTTP 429s with multi-provider LLM fallback codeStep-by-step guide to handle 429 rate limit llm fallback across multiple LLM providers with practical Python code, backoff, circuit breakers, and verification.API integrationHow to chain three LLM providers as automatic fallbacksLearn how to chain LLM providers with automatic fallback in Python, using OpenAI-compatible endpoints and robust error handling for production.

All multi-provider fallback code patterns posts →