Graceful error responses for LLM failures in Echo

When you wire LLM calls into an Echo service, naive error propagation turns provider outages into cryptic 500s. Robust echo error handling llm failures means mapping upstream timeouts, rate limits, and malformed outputs into structured responses your clients can act on. This guide lays out an ordered path to make that happen without bloating your handlers.

1. Define a contract for LLM errors

Start by deciding what the client receives. A generic {"error": "internal"} forces callers to guess. Define a typed error envelope:

type LLMError struct {
    Code      string `json:"code"`
    Message   string `json:"message"`
    Retryable bool   `json:"retryable"`
    TraceID   string `json:"trace_id,omitempty"`
}

Use stable codes: provider_timeout, rate_limited, bad_request, upstream_unavailable. Clients script against these, not against HTTP status alone.

Tradeoff: adding fields later breaks strict consumers. Version your envelope or document backward compatibility from day one.

2. Centralize error handling with Echo middleware

Echo’s HTTPErrorHandler is the single place to convert panics and returned errors into JSON. Don’t call c.JSON in every route.

e.HTTPErrorHandler = func(err error, c echo.Context) {
    var he *echo.HTTPError
    if errors.As(err, &he) {
        // already wrapped by a handler
    } else {
        he = echo.NewHTTPError(http.StatusInternalServerError, err.Error())
    }
    resp := LLMError{
        Code:      codeFromHTTP(he.Code),
        Message:   fmt.Sprintf("%v", he.Message),
        Retryable: he.Code >= 500,
        TraceID:   c.Response().Header().Get(echo.HeaderXRequestID),
    }
    c.JSON(he.Code, resp)
}

Pitfall: returning the raw err.Error() leaks stack traces if you forget to wrap. Always map to your envelope before serialization.

3. Classify upstream failures explicitly

Your LLM client should return sentinel errors, not strings. Define them once:

var (
    ErrProviderTimeout = errors.New("provider timeout")
    ErrRateLimited     = errors.New("rate limited")
    ErrBadRequest      = errors.New("bad request to provider")
    ErrTruncated       = errors.New("truncated output")
)

When calling the model inside a handler:

resp, err := client.Complete(ctx, prompt)
if err != nil {
    switch {
    case errors.Is(err, ErrProviderTimeout):
        return echo.NewHTTPError(http.StatusGatewayTimeout, err)
    case errors.Is(err, ErrRateLimited):
        return echo.NewHTTPError(http.StatusTooManyRequests, err)
    case errors.Is(err, ErrBadRequest):
        return echo.NewHTTPError(http.StatusBadRequest, err)
    case errors.Is(err, ErrTruncated):
        return echo.NewHTTPError(http.StatusUnprocessableEntity, err)
    default:
        return err
    }
}

This keeps echo error handling llm failures mechanical: the middleware just serializes.

Watch for context cancellation

If the client disconnects, ctx.Err() is context.Canceled. Return 499 (or let Echo’s default handle it) and don’t log it as a provider fault. Misclassifying cancellation as a 5xx inflates your incident metrics.

4. Implement retry and fallback logic

Retries belong in the LLM client, not the HTTP layer. Use a bounded backoff with jitter:

func CompleteWithRetry(ctx context.Context, c *Client, prompt string) (string, error) {
    var lastErr error
    for i := 0; i < 3; i++ {
        resp, err := c.Complete(ctx, prompt)
        if err == nil {
            return resp, nil
        }
        if !errors.Is(err, ErrProviderTimeout) && !errors.Is(err, ErrRateLimited) {
            return "", err // do not retry 4xx-class
        }
        lastErr = err
        select {
        case <-ctx.Done():
            return "", ctx.Err()
        case <-time.After(backoff(i)):
        }
    }
    return "", lastErr
}

If you route through a gateway such as n4n.ai, it performs automatic fallback when a provider is rate-limited or degraded; still, your Echo handler must detect the forwarded error and respond gracefully. Don’t double-retry across the gateway boundary unless you understand the added latency and cost.

Tradeoff: retries amplify load during incidents. Cap concurrency per handler and use jitter so clients don’t synchronize their backoffs.

5. Return structured responses with correct status codes

Map each error class to a status. A rate limit must be 429 with a Retry-After header. Set it before returning the HTTP error:

if errors.Is(err, ErrRateLimited) {
    c.Response().Header().Set("Retry-After", "2")
    return echo.NewHTTPError(http.StatusTooManyRequests, err)
}

Clients using fetch or axios can read Retry-After and back off. Without it, they hammer your endpoint and deepen the outage.

Don’t 200 with an error body

Some LLM SDKs return partial JSON on truncated output. That’s not an HTTP transport error but a semantic one. Return 422 with code: "truncated_output" so the caller can re-prompt or trim context.

Example envelope on truncation:

{
  "code": "truncated_output",
  "message": "completion exceeded max tokens",
  "retryable": false,
  "trace_id": "req-8f1c"
}

6. Log and meter without leaking secrets

Log the trace_id and error class, not the prompt or token counts if they contain PII. Use Echo’s request ID:

c.Logger().Errorf("llm call failed: code=%s trace=%s", code, c.Response().Header().Get(echo.HeaderXRequestID))

If you meter per-token usage, do it in a deferred function and write to a metrics pipe, not the error response. Exposing usage on failure is noise and can leak context window sizes.

Pitfall: logging the full upstream response can capture system prompts you reused across tenants. Redact before the line hits the writer.

7. Test failure paths with httptest

Write table tests that inject each sentinel error and assert the envelope:

func TestErrorEnvelope(t *testing.T) {
    e := echo.New()
    e.HTTPErrorHandler = handler
    // serve with fake client returning ErrRateLimited
    req := httptest.NewRequest("POST", "/complete", nil)
    rec := httptest.NewRecorder()
    e.ServeHTTP(rec, req)
    if rec.Code != http.StatusTooManyRequests {
        t.Fatalf("got %d", rec.Code)
    }
    var body LLMError
    json.Unmarshal(rec.Body.Bytes(), &body)
    if body.Code != "rate_limited" {
        t.Fatalf("bad code: %s", body.Code)
    }
}

Echo error handling llm failures is only proven if CI fails when you regress the contract. Run these tests on every PR.

Simulate provider degradation

Run a local proxy that returns 503 for 10% of calls. Verify your retry and fallback don’t cascade into a self-inflicted denial of service.

8. Handle streaming failures separately

If you stream tokens via text/event-stream, you can’t send a JSON error after headers flush. Emit a final SSE event with your envelope:

fmt.Fprintf(w, "event: error\ndata: %s\n\n", jsonEnvelope)

The client must parse SSE and handle the error event. Tradeoff: Echo middlewares don’t run post-flush; you must detect write errors inline and close the stream cleanly. Don’t attempt to send a trailing HTTP status—the connection is already committed.

Common pitfalls and tradeoffs

• Over-wrapping: converting every error to 500 loses signal. Preserve 4xx vs 5xx distinctions.
• Leaky abstractions: returning the provider’s raw error string exposes their internal versioning and confuses clients.
• Synchronous retries on the request path: blocks the Echo worker. Offload long retries to a queue if latency budgets are tight.
• Ignoring partial success: a stream that dies at token 500 needs a different client action than a hard 502.

Good echo error handling llm failures treats the LLM as a flaky network service, not a local library call. Build the envelope once, classify at the boundary, and let middleware do the boring serialization.