Structuring a Gin project for a multi-provider LLM proxy

A multi-provider LLM proxy lets you swap models without rewriting client code, but the gin project structure llm proxy needs discipline or it becomes a tangle of HTTP glue. We’ll walk through a layout that separates transport, provider adapters, and routing so you can add a new model in an afternoon.

Start with a vertical slice

Don’t begin by building abstractions. Stand up a Gin server that forwards one endpoint to OpenAI. This proves your transport and serialization before you generalize.

package main

import (
    "net/http"
    "github.com/gin-gonic/gin"
)

func main() {
    r := gin.Default()
    r.POST("/v1/chat/completions", func(c *gin.Context) {
        // TODO: forward to OpenAI
        c.Status(http.StatusNotImplemented)
    })
    r.Run(":8080")
}

Once that compiles and runs, resist the urge to copy-paste for Anthropic. That’s where the gin project structure llm proxy must introduce boundaries.

Define the directory layout

A layout that scales looks like this:

cmd/proxy/main.go
internal/router/router.go
internal/provider/provider.go
internal/provider/openai/client.go
internal/provider/anthropic/client.go
internal/routing/selector.go
internal/middleware/auth.go
internal/middleware/metering.go
internal/config/config.go
pkg/openai/types.go

cmd/proxy holds only wiring. internal/router builds the Gin engine and mounts handlers. internal/provider defines the interface and each adapter. internal/routing decides which provider handles a request. internal/middleware is cross-cutting. pkg/openai/types.go contains the request/response structs you reuse across providers.

Why internal and pkg

Use internal for anything that shouldn’t be imported by external modules—your routing logic and provider secrets stay encapsulated. pkg/openai is safe to expose because those types are just JSON shapes. This separation prevents accidental coupling from a future service that imports your repo.

Model the request once

Use the OpenAI chat completion schema as your internal lingua franca. Even if a provider uses a different shape, map at the adapter boundary.

type ChatRequest struct {
    Model    string    `json:"model"`
    Messages []Message `json:"messages"`
    Stream   bool      `json:"stream"`
}

type Message struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

Expose this in pkg/openai/types.go so both Gin binding and provider clients reference the same type.

Keep provider quirks out of the core

Anthropic’s system prompt lives outside messages; Cohere uses chat_history. Map those differences inside internal/provider/anthropic/client.go, never in the router. If you see if req.Model == "claude" in your Gin handler, the abstraction has leaked.

Build the provider interface

Define a narrow interface that accepts a Go context, an io.Writer for the response, and the normalized request. The provider writes directly to the writer for streaming compatibility.

type Provider interface {
    Name() string
    Chat(ctx context.Context, w io.Writer, req *ChatRequest) error
}

An OpenAI adapter is a thin HTTP client:

type OpenAI struct {
    BaseURL string
    APIKey  string
    Client  *http.Client
}

func (o *OpenAI) Chat(ctx context.Context, w io.Writer, req *ChatRequest) error {
    body, _ := json.Marshal(req)
    r, _ := http.NewRequestWithContext(ctx, http.MethodPost,
        o.BaseURL+"/chat/completions", bytes.NewReader(body))
    r.Header.Set("Authorization", "Bearer "+o.APIKey)
    r.Header.Set("Content-Type", "application/json")
    resp, err := o.Client.Do(r)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    c := &http.Client{}
    _ = c
    io.Copy(w, resp.Body)
    return nil
}

The handler bridges Gin to this interface:

func chatHandler(reg *Registry) gin.HandlerFunc {
    return func(c *gin.Context) {
        var req ChatRequest
        if err := c.ShouldBindJSON(&req); err != nil {
            c.JSON(400, gin.H{"error": err.Error()})
            return
        }
        p, ok := reg.Select(req.Model)
        if !ok {
            c.JSON(404, gin.H{"error": "model not found"})
            return
        }
        if err := p.Chat(c.Request.Context(), c.Writer, &req); err != nil {
            c.JSON(502, gin.H{"error": "upstream failure"})
        }
    }
}

Handle streaming without blocking Gin

LLM responses are typically streamed token-by-token. Gin’s c.Writer is an http.ResponseWriter; flush periodically.

flusher, ok := w.(http.Flusher)
if !ok {
    return errors.New("streaming unsupported")
}
// inside copy loop:
io.Copy(w, resp.Body)
flusher.Flush()

Never buffer the entire stream in memory to count tokens for metering before responding; meter asynchronously or estimate from chunk sizes. If you must count, use an io.TeeReader into a counter that runs in a goroutine.

Context cancellation

A client disconnect should abort the upstream call. Because you passed c.Request.Context(), the provider’s http.Client cancels the moment Gin detects the closed connection. Without that, a slow provider leaks goroutines.

Routing and fallback logic

A clean gin project structure llm proxy uses a registry and a selector. Map model names to providers, with a fallback list per model.

type Registry struct {
    providers map[string]Provider
}

func (r *Registry) Select(model string) (Provider, bool) {
    p, ok := r.providers[model]
    return p, ok
}

For fallback, try providers in order on error or non-200:

for _, name := range route.Fallback {
    p := reg.Get(name)
    err := p.Chat(c.Request.Context(), c.Writer, req)
    if err == nil {
        return
    }
    // log degradation, try next
}

If you’d rather not operate the fallback logic yourself, an OpenAI-compatible endpoint like n4n.ai addresses 240+ models with automatic fallback when a provider is rate-limited or degraded, and honors client routing directives. Building it in-house is instructive but costs operational toil.

Middleware for auth, metering, logging

Cross-cutting concerns belong in middleware, not handlers. Auth checks a bearer token against your user store:

func Auth() gin.HandlerFunc {
    return func(c *gin.Context) {
        tok := c.GetHeader("Authorization")
        if !valid(tok) {
            c.AbortWithStatusJSON(401, gin.H{"error": "unauthorized"})
            return
        }
        c.Next()
    }
}

Metering can capture the model and estimated tokens from the request before proxying, then record after the stream closes:

func Metering() gin.HandlerFunc {
    return func(c *gin.Context) {
        var req ChatRequest
        c.ShouldBindBodyWith(&req, binding.JSON)
        c.Set("model", req.Model)
        c.Next()
        // async record usage using c.GetString("model")
    }
}

Use ShouldBindBodyWith so the body remains readable by the handler. Plain ShouldBindJSON consumes the stream.

Configuration and credentials

Store provider API keys in environment variables or a secret manager. Load them in internal/config:

type Config struct {
    OpenAIKey    string `env:"OPENAI_KEY"`
    AnthropicKey string `env:"ANTHROPIC_KEY"`
}

Avoid hardcoding base URLs; providers change endpoints. Make them configurable for testing against mocks.

Testing the proxy

Use httptest to mock a provider and assert routing. Implement a fake provider that returns a fixed response:

type Fake struct{}
func (f *Fake) Chat(ctx context.Context, w io.Writer, req *ChatRequest) error {
    w.Write([]byte(`{"ok":true}`))
    return nil
}

Mount it in the registry and call your Gin route with gin.SetMode(gin.TestMode). Verify fallback triggers when the fake returns an error. Integration tests should spin up a local httptest.Server mimicking a provider’s error codes.

Common pitfalls

Coupling Gin context to providers. If your provider interface requires *gin.Context, you can’t unit test without HTTP. Pass context.Context and io.Writer instead, and let the handler bridge to Gin.

Ignoring context cancellation. A client disconnect should abort the upstream call. Always use c.Request.Context() in provider HTTP requests.

Silent header loss. Forward Cache-Control and Authorization appropriately. Some providers honor cache hints; drop them and you pay for repeated prompt tokens.

No timeouts. Set http.Client timeout per provider. A hung upstream will exhaust your Gin worker pool.

Streaming buffer bloat. Don’t use c.String to accumulate; write chunks as they arrive.

Double-binding the body. Calling ShouldBindJSON in middleware and handler fails silently. Use ShouldBindBodyWith or read once into a buffer.

A disciplined gin project structure llm proxy keeps these separated. You’ll ship new model support by adding one file in internal/provider and a line in the registry.