Building a multi-provider integration in Go starts with boundaries. A clean go package structure llm client separates the provider-agnostic API from the messy HTTP details of each backend, so you can add Anthropic or Mistral without rewriting call sites. This guide walks through an ordered path from interface definition to on-disk layout, with the tradeoffs we hit shipping similar clients.
1. Define a provider-agnostic core interface
Start by writing the smallest interface that every backend must satisfy. Keep it free of any vendor-specific fields. If you leak temperature or top_p into the root type, you will regret it when a provider uses a different parameter shape.
package llm
import "context"
type Message struct {
Role string // "system", "user", "assistant"
Content string
}
type Usage struct {
PromptTokens int
CompletionTokens int
}
type ChatRequest struct {
Model string
Messages []Message
}
type Response struct {
ID string
Model string
Message Message
Usage Usage
}
type Client interface {
Chat(ctx context.Context, req ChatRequest) (*Response, error)
}This core gives call sites a stable target. Adapters map their wire format to these structs. The interface is deliberately narrow; streaming and advanced options live elsewhere.
2. Isolate provider specifics in adapter packages
The go package structure llm client we advocate puts each provider in its own subpackage under the root. The openai package implements llm.Client; the anthropic package does the same. Neither imports the other.
package openai
import (
"context"
"net/http"
"yourmodule/llm"
)
type Client struct {
baseURL string
apiKey string
http *http.Client
}
func New(baseURL, apiKey string, h *http.Client) *Client {
return &Client{baseURL: baseURL, apiKey: apiKey, http: h}
}
func (c *Client) Chat(ctx context.Context, req llm.ChatRequest) (*llm.Response, error) {
// translate req to OpenAI's JSON, POST, decode, translate back
}Tradeoff: you duplicate the request-mapping logic per provider. That is cheaper than a tangled union type that breaks when either vendor changes. Keep the translation pure where possible—no context or http inside the mapper functions.
3. Build a routing and fallback layer
A multi-provider system needs a component that decides which adapter handles a call. Implement the same llm.Client interface so the router is transparent to business code.
package router
type Router struct {
primary llm.Client
fallback llm.Client
}
func (r *Router) Chat(ctx context.Context, req llm.ChatRequest) (*llm.Response, error) {
resp, err := r.primary.Chat(ctx, req)
if err != nil {
return r.fallback.Chat(ctx, req)
}
return resp, nil
}Error classification matters: only fall back on 429/5xx, not on a malformed prompt. If you would rather not operate the fallback logic yourself, a gateway such as n4n.ai exposes a single OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded, but self-hosted systems still need this pattern for control over routing policy.
4. Configure transport with net/http explicitly
Never use http.DefaultClient in a long-lived service. Set timeouts and connection pooling once, then inject the client into every adapter.
transport := &http.Transport{
MaxIdleConns: 100,
MaxIdleConnsPerHost: 10,
IdleConnTimeout: 90 * time.Second,
}
httpClient := &http.Client{
Timeout: 30 * time.Second,
Transport: transport,
}A missing Timeout lets a stalled TCP connection block a goroutine indefinitely. Connection reuse cuts TLS handshake cost on high-throughput batch jobs.
5. Handle streaming as a separate path
Streaming responses do not fit a (*Response, error) signature. Add a distinct interface rather than overloading Chat.
type StreamChunk struct {
Delta string
Done bool
}
type StreamingClient interface {
StreamChat(ctx context.Context, req ChatRequest) (<-chan StreamChunk, error)
}Adapters implement StreamingClient only if the provider supports SSE. Call sites that need tokens incrementally use the stream; batch callers stay on Chat. The tradeoff is two code paths per provider, but the alternative—returning io.ReadCloser from Chat—forces non-streaming users to drain a body they did not ask for.
6. Compose request options functionally
Advanced parameters vary wildly. Use the functional options pattern at the adapter level, not in the core ChatRequest.
type Option func(*requestOpts)
type requestOpts struct {
temperature float32
maxTokens int
}
func WithTemperature(t float32) Option {
return func(o *requestOpts) { o.temperature = t }
}
func (c *Client) ChatWithOpts(ctx context.Context, req llm.ChatRequest, opts ...Option) (*llm.Response, error) {
o := &requestOpts{temperature: 0.7, maxTokens: 512}
for _, opt := range opts {
opt(o)
}
// merge o into provider payload
}This keeps the root package stable while letting each backend expose its quirks. Avoid putting Option in llm.Client; it would infect every adapter with the union of all providers’ knobs.
7. Lay out the package on disk
A concrete tree prevents “where does this go” debates:
llm/
client.go # interfaces and core types
types.go # Message, Usage, etc.
router/
router.go # primary/fallback orchestration
openai/
client.go
mapper.go # req/resp translation
anthropic/
client.go
mapper.go
internal/
httpx/
client.go # shared transport helpersThis go package structure llm client keeps the root package small and makes the dependency direction obvious: adapters depend on llm, never on each other. internal/httpx holds retry and auth header logic reused across adapters.
8. Test against recorded fixtures
Use httptest to simulate provider APIs. Record real responses once, replay them in unit tests.
func TestOpenAIAdapter(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
_ = json.NewEncoder(w).Encode(fixtureOpenAIResponse)
}))
defer ts.Close()
c := openai.New(ts.URL, "test-key", http.DefaultClient)
resp, err := c.Chat(context.Background(), llm.ChatRequest{Model: "gpt-4o"})
if err != nil {
t.Fatal(err)
}
if resp.Usage.CompletionTokens == 0 {
t.Fatalf("expected token accounting")
}
}Test the router’s fallback by making the primary server return 500. Engineers often skip this and discover fallback bugs in production when a provider actually degrades.
9. Common pitfalls and tradeoffs
- Context propagation: pass
ctxinto every HTTP call and honor cancellation. Adapters that ignorectxwill leak goroutines during client timeouts. - Token metering: normalize usage into
llm.Usageeven if a provider returns a different schema. Downstream billing code should not know provider names. - Version drift: provider APIs change weekly. Isolate the JSON tags in
mapper.goso a field rename touches one file. - Over-abstraction: do not build a generic “model capability” matrix in the core. Let the router take plain
llm.Clientinstances; capability filtering belongs in config. - Sync vs async: blocking
Chatis correct for most RPC-style calls. Forcing async everywhere adds goroutine bookkeeping with no benefit.
A disciplined go package structure llm client treats providers as pluggable adapters behind a tiny interface, puts transport configuration in one place, and reserves streaming and options for explicit secondary paths. That layout survives adding the tenth provider without a rewrite.