Error handling for LLM API clients in Rust with thiserror

Most Rust LLM API clients ship with error handling that either panics on unexpected JSON or exposes raw reqwest::Error to callers. Good rust thiserror llm api error handling means defining a typed surface that separates transport failures from provider rejections and decode errors, so your application can decide whether to retry, fall back, or bail. This guide walks through a concrete pattern we use for production clients.

1. Model the failure modes explicitly

Start by declaring one enum that represents every way the client can fail at the boundary. Do not let reqwest or serde errors escape directly—those are implementation details. thiserror gives you #[derive(Error)] and clean Display impls without boilerplate.

use thiserror::Error;

#[derive(Error, Debug)]
pub enum LlmClientError {
    #[error("transport failure: {0}")]
    Transport(#[from] reqwest::Error),

    #[error("failed to decode response: {0}")]
    Decode(#[from] serde_json::Error),

    #[error("provider returned status {status}: {message}")]
    Provider {
        status: u16,
        message: String,
        request_id: Option<String>,
    },

    #[error("rate limited, retry after {retry_after:?}")]
    RateLimited { retry_after: Option<u64> },

    #[error("authentication failed: {0}")]
    Auth(String),

    #[error("unknown error: {0}")]
    Unknown(String),
}

The #[from] attributes are deliberate. We only forward underlying errors that are unambiguous: a transport failure is a transport failure, and a JSON decode error is a decode error. Provider HTTP errors get mapped to Provider or RateLimited variants via status codes, not via #[from] reqwest::Response.

A common mistake is adding #[from] for every dependency error type. If two dependencies share an error type, or if you later add a streaming client with its own error, the conflicting From impls will break compilation. Keep the enum small and map manually where ambiguity exists.

2. Convert HTTP responses into typed errors

Your request method should inspect the status code before attempting to deserialize a success body. The error path needs to parse the provider’s error envelope, which for OpenAI-compatible APIs is typically:

{ "error": { "message": "invalid api key", "type": "invalid_request_error" } }

Define a minimal struct for that shape and convert:

#[derive(serde::Deserialize)]
struct ProviderError {
    error: InnerError,
}

#[derive(serde::Deserialize)]
struct InnerError {
    message: String,
}

async fn check_status(resp: reqwest::Response) -> Result<reqwest::Response, LlmClientError> {
    let status = resp.status();
    if status.is_success() {
        return Ok(resp);
    }
    let body = resp.text().await.unwrap_or_default();
    if status == reqwest::StatusCode::TOO_MANY_REQUESTS {
        let retry_after = resp
            .headers()
            .get("retry-after")
            .and_then(|v| v.to_str().ok())
            .and_then(|s| s.parse().ok());
        return Err(LlmClientError::RateLimited { retry_after });
    }
    if status == reqwest::StatusCode::UNAUTHORIZED {
        return Err(LlmClientError::Auth(body));
    }
    let msg = serde_json::from_str::<ProviderError>(&body)
        .map(|e| e.error.message)
        .unwrap_or(body);
    Err(LlmClientError::Provider {
        status: status.as_u16(),
        message: msg,
        request_id: resp
            .headers()
            .get("x-request-id")
            .map(|v| v.to_string()),
    })
}

Note the unwrap_or(body) fallback. Some gateways return plain text on 502s. If you assume JSON only, you turn a provider outage into a Decode error and lose the status code.

3. Classify retryability at the error level

Applications need to know whether to back off or surface the error to a user. Add a method, not a trait, to keep it local:

impl LlmClientError {
    pub fn is_retryable(&self) -> bool {
        match self {
            LlmClientError::Transport(e) => e.is_timeout() || e.is_connect(),
            LlmClientError::RateLimited { .. } => true,
            LlmClientError::Provider { status, .. } => *status >= 500,
            _ => false,
        }
    }
}

Tradeoff: a 500 from a provider can mean either a transient backend hiccup or a malformed request that the model server rejected after auth. Retrying blindly wastes quota. In practice we cap retries at three and only retry Provider errors when the status is 502/503/504, not 500. Adjust the match to your provider’s observed behavior.

4. Handle streaming errors as a distinct path

If you expose a streaming completion API (SSE or chunked JSON), the connection can fail after the first byte. A reqwest::Stream error mid-flight should not be silently swallowed by Drop. Define a separate LlmStreamError or add a Stream variant to the same enum:

#[error("stream terminated unexpectedly at byte {offset}")]
Stream { offset: usize, source: reqwest::Error },

The pitfall here is that futures::StreamExt::next() returns None on both clean EOF and underlying error if you use the wrong combinator. Use reqwest::Response::bytes_stream() and map each Result<Bytes, reqwest::Error> explicitly so transport failures become LlmClientError::Stream.

5. Keep provider specifics out of the public API

An inference gateway like n4n.ai normalizes many backends behind one OpenAI-compatible endpoint, but your client should still map the unified error shape rather than assume a single vendor’s quirks. That means your LlmClientError stays stable even if you switch from a direct OpenAI connection to a routed gateway that adds x-request-id or translates rate-limit headers. The client code above already captures request_id optionally, so it works whether the header is present or not.

6. Choose the right error boundary

For a library crate, thiserror is the correct choice: callers get a concrete enum they can match on. For a binary or service, wrapping with anyhow at the top level is fine, but do not let anyhow::Error leak into the library surface.

// inside a binary's main
let reply = match client.complete(req).await {
    Ok(r) => r,
    Err(e) => {
        tracing::error!(error = %e, "llm call failed");
        return Err(anyhow::anyhow!(e));
    }
};

If you use Box<dyn std::error::Error> in your public signatures, you force every caller to string-match on Display output. That is worse than no typed errors at all.

7. Add observability context without bloating

Attach the fields you will need in logs: model name, request id, latency. Put them in the variant or in a wrapper struct:

#[derive(Error, Debug)]
#[error("request to {model} failed: {source}")]
pub struct InstrumentedError {
    model: String,
    request_id: Option<String>,
    #[source]
    source: LlmClientError,
}

This preserves the underlying error chain for anyhow/tracing while giving structured fields. Do not stuff ten strings into Unknown(String)—that is just a disguised untyped error.

8. Test the error mapping

Error handling is code; treat it like code. Stand up a local mock that returns canned statuses and assert the variant:

#[tokio::test]
async fn maps_429_to_rate_limited() {
    let mock = wiremock::MockServer::start().await;
    wiremock::Mock::given(wiremock::matchers::method("POST"))
        .respond_with(
            wiremock::ResponseTemplate::new(429)
                .insert_header("retry-after", "2"),
        )
        .mount(&mock)
        .await;

    let client = LlmClient::new(mock.uri());
    let err = client.complete(Default::default()).await.unwrap_err();
    assert!(matches!(
        err,
        LlmClientError::RateLimited { retry_after: Some(2) }
    ));
}

If you skip this, the first time you see a 429 in production it will probably be mapped to Unknown because the header parse silently failed.

Common pitfalls

Over-broad #[from]: deriving From for both std::io::Error and reqwest::Error when a helper returns either will cause conflicting impls. Map manually.
Assuming JSON error bodies: a 502 from a load balancer is often HTML. Use unwrap_or_else to keep the raw body.
Losing retry state: returning RateLimited { retry_after: None } when the header is missing forces the caller to guess. Default to a sane constant (e.g., 5s) at the call site, not inside the enum.
Ignoring stream teardown: a dropped SSE stream can panic in async runtimes if the body is not polled to completion. Always while let Some(chunk) = stream.next().await and handle Err.

Implementation checklist

Define LlmClientError with thiserror, separating transport, decode, provider, rate-limit, auth.
Write one check_status function that converts non-2xx responses into the right variant.
Implement is_retryable and document the retry cap.
Add a streaming error variant if you expose streams.
Keep the enum provider-agnostic; capture request_id optionally.
Use anyhow only at the process boundary.
Write mock tests for 401, 429, 500, and non-JSON 502.

Following this path makes rust thiserror llm api error handling a first-class part of your client instead of an afterthought, and your callers will spend less time debugging reqwest::Error chains at 3 a.m.