Error handling patterns for Go LLM API clients

Building a reliable go error handling llm api client in Go requires more than checking a non-nil error. LLM providers fail in ways traditional HTTP services do not: partial streaming responses, rate limits, and schema drift demand explicit handling at the boundary.

1. Model failures as typed errors

A bare error from http.Client tells you nothing about whether the request hit the network, the provider rejected your schema, or the model hallucinated a malformed token. Define a closed set of error kinds so callers can branch.

type ErrorKind int

const (
    KindTransport ErrorKind = iota // TCP/DNS/TLS failure
    KindTimeout
    KindHTTPStatus // 4xx/5xx
    KindAPI        // 200 but error field set
    KindTruncated  // stream cut off
)

type LLMError struct {
    Kind      ErrorKind
    Status    int
    Retryable bool
    Msg       string
    Err       error
}

func (e *LLMError) Error() string { return e.Msg }
func (e *LLMError) Unwrap() error { return e.Err }

Map every failure at the client boundary into *LLMError. Callers use errors.As to decide behavior. This avoids the Python-style catch-all that hides whether a retry is safe.

Pitfall: wrapping too early

Do not wrap raw json.SyntaxError as a generic error before classifying it. Keep the low-level error in Err and set Kind explicitly.

2. Bound every request with context timeouts

LLM inference latency has a long tail. A missing timeout hangs a goroutine indefinitely. Set a deadline on the request context, not just the http.Server.

ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()

req, _ := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, body)
resp, err := client.Do(req)
if ctx.Err() == context.DeadlineExceeded {
    return &LLMError{Kind: KindTimeout, Retryable: true, Msg: "request timed out"}
}

Tune http.Transport separately for TLS handshake and idle conns. A go error handling llm api client should also set Transport.ResponseHeaderTimeout to fail fast on stalled connections.

Tradeoff: too-short timeouts

Small models return in <200ms; large ones take minutes. Use a per-model timeout map rather than a single constant.

3. Retry only idempotent, retryable errors

Retrying a non-idempotent request can double-bill tokens or produce duplicate side effects. Classify:

• KindTransport, KindTimeout, 429, 500, 503 → retryable
• 400, 401, 422 → non-retryable (bad input)
• KindTruncated on a streaming completion → retryable only if you can discard partial output

Use exponential backoff with jitter. The golang.org/x/time/rate or a small custom loop suffices.

func doWithRetry(ctx context.Context, client *http.Client, req *http.Request) (*http.Response, error) {
    var last *LLMError
    for attempt := 0; attempt < 3; attempt++ {
        resp, err := client.Do(req)
        if err == nil && resp.StatusCode < 500 {
            return resp, nil
        }
        if resp != nil && resp.StatusCode == 429 {
            backoff := time.Duration(1<<attempt) * 100 * time.Millisecond
            select {
            case <-time.After(backoff):
            case <-ctx.Done():
                return nil, ctx.Err()
            }
            continue
        }
        // wrap and maybe break
    }
    return nil, last
}

Common mistake: retrying on 401

A 401 means your API key is wrong. Retrying wastes quota and leaks credentials to logs.

4. Handle streaming errors mid-response

Chat completions often stream via SSE. The HTTP status is 200, but the body can truncate when a provider crashes mid-generation. Detect this by tracking whether the final [DONE] sentinel arrived.

scanner := bufio.NewScanner(resp.Body)
var gotDone bool
for scanner.Scan() {
    line := scanner.Text()
    if line == "data: [DONE]" {
        gotDone = true
        break
    }
    // parse JSON chunk
}
if !gotDone {
    return &LLMError{Kind: KindTruncated, Retryable: true, Msg: "stream ended without [DONE]"}
}

If you buffer the whole response, check resp.Body.Close() error and scanner error separately.

Tradeoff: partial output usability

Sometimes a truncated response is still useful. Expose KindTruncated but return the partial text so the caller can decide to degrade gracefully.

5. Implement provider fallback without leaking state

When a primary model is degraded, you may call a secondary. If you route through an OpenAI-compatible gateway like n4n.ai, it automatically falls back when a provider is rate-limited or degraded, but your client still owns timeout and typed-error mapping. If you hand-roll fallback, clone the request body because http.Request.Body is single-use.

bodyBytes, _ := io.ReadAll(req.Body)
for _, endpoint := range fallbacks {
    req, _ = http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(bodyBytes))
    resp, err := client.Do(req)
    if err == nil && resp.StatusCode == 200 {
        return resp, nil
    }
}

Never reuse the same context.WithTimeout across fallbacks without extending it; the original deadline will expire mid-fallback.

6. Surface rate-limit and usage metadata

Providers return headers like x-ratelimit-remaining and JSON usage objects. A robust go error handling llm api client parses these to drive local throttling.

remaining := resp.Header.Get("x-ratelimit-remaining")
if remaining == "0" {
    // back off locally before next call
}

If the gateway forwards per-token usage metering, capture it for cost attribution:

var out struct {
    Usage struct {
        PromptTokens     int `json:"prompt_tokens"`
        CompletionTokens int `json:"completion_tokens"`
    } `json:"usage"`
}
json.Unmarshal(respBytes, &out)

Pitfall: ignoring 429 Retry-After

The Retry-After header may be in seconds or an HTTP date. Parse both; ignoring it causes thundering retries.

7. Test error paths with httptest

Use httptest.Server to simulate 500s, slow responses, and malformed JSON. Table-driven tests catch missing classifications.

func TestClient_Errors(t *testing.T) {
    srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(500)
    }))
    defer srv.Close()
    // assert KindHTTPStatus, Retryable true
}

Inject a context with a 1ms timeout to verify KindTimeout mapping. This is cheaper than integration tests against live providers.

Tradeoff: mock fidelity

httptest won’t replicate provider-specific streaming quirks. Keep one golden integration test behind a build tag.

Common pitfalls and tradeoffs summary

• Over-wrapping: wrapping errors without kind loses retryability signals.
• Shared body: reusing req.Body across retries fails silently; read once.
• Log noise: logging full prompt on every 429 leaks PII; log kind and status only.
• Timeout coupling: a 30s timeout on a 3-model fallback chain means the last model gets 0s.

A disciplined go error handling llm api client treats the network as hostile and the model as unreliable. Typed errors, bounded contexts, and explicit retry rules turn intermittent provider outages into manageable events.