n4nAI
Open console →
API integrationHow-toRust LLM API Client

Retry logic for Rust LLM API clients

Practical guide to building robust rust llm api retry logic with exponential backoff, jitter, and correct error classification for LLM clients.

n4n Team3 min read596 words

Audio narration

Coming soon — every post will get a voice note here.

A naive HTTP retry loop will burn tokens and mask real failures when you point it at a language model endpoint. Solid rust llm api retry logic separates transient transport errors from semantic API rejections, and backs off without stampeding the server.

Step 1: Scaffold the project and pull in HTTP + backoff crates

Create a fresh binary crate. We’ll use reqwest with rustls to avoid OpenSSL linkage, tokio for the async runtime, backoff for retry primitives, and serde for payloads. Keep the dependency surface small; you don’t need a heavy LLM SDK to get correct behavior.

cargo new llm_retry && cd llm_retry

Add this to Cargo.toml:

[dependencies]
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
backoff = { version = "0.4", features = ["tokio"] }
rand = "0.8"
uuid = { version = "1", features = ["v4"] }
anyhow = "1"

The rust llm api retry logic we build on top of these crates stays independent of any specific model vendor. Set a client timeout early so a hung connection doesn’t block the retry state machine:

let client = reqwest::Client::builder()
    .timeout(std::time::Duration::from_secs(20))
    .build()?;

Step 2: Classify errors so you retry the right things

LLM endpoints return a mix of transport failures and HTTP status codes. Retrying a 400 is pointless; retrying a 429 or 503 is mandatory. Write an error type that maps reqwest::Error and StatusCode into a retry decision.

use reqwest::StatusCode;
use std::fmt;

#[derive(Debug)]
pub enum LlmError {
    Transport(reqwest::Error),
    RateLimited(Option<u64>),
    Server(u16),
    Client(u16, String),
    Auth,
}

impl fmt::Display for LlmError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{:?}", self)
    }
}

pub fn classify(status: StatusCode, err: Option<reqwest::Error>) -> LlmError {
    if let Some(e) = err {
        return LlmError::Transport(e);
    }
    match status {
        StatusCode::TOO_MANY_REQUESTS => {
            let retry_after = status
                .headers()
                .get("retry-after")
                .and_then(|v| v.to_str().ok())
                .and_then(|s| s.parse().ok());
            LlmError::RateLimited(retry_after)
        }
        StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN => LlmError::Auth,
        s if s.is_server_error() => LlmError::Server(s.as_u16()),
        s if s.is_client_error() => LlmError::Client(s.as_u16(), "bad request".into()),
        _ => LlmError::Client(status.as_u16(), "unexpected".into()),
    }
}

If you route through a gateway such as n4n.ai, which performs automatic fallback when a provider is rate-limited or degraded, your client retry can focus on 5xx and connection resets rather than provider-specific 429 nuances. The classification above still works; you just see fewer rate-limit errors reaching the client.

Step 3: Configure exponential backoff with jitter

The core of rust llm api retry logic is a backoff schedule that grows and adds randomness to avoid thundering herds. Use backoff::ExponentialBackoff and apply full jitter.

use backoff::ExponentialBackoff;

fn backoff_policy() -> ExponentialBackoff {
    let mut b = ExponentialBackoff {
        initial_interval: std::time::Duration::from_millis(200),
        max_interval: std::time::Duration::from_secs(8),
        multiplier: 2.0,
        max_elapsed_time: Some(std::time::Duration::from_secs(30)),
        ..Default::default()
    };
    // Full jitter: delay is uniformly random in [0, computed]
    b.randomization_factor = 1.0;
    b
}

randomization_factor = 1.0 makes the actual sleep uniformly distributed between 0 and the computed bound. For a single client this is enough. If you run many workers, equal-jitter (delay/2 + random/2) can reduce median latency; tune the factor between 0 and 1 accordingly. Always cap max_elapsed_time so a permanently broken upstream fails fast instead of hanging for minutes.

Step 4: Wrap the API call in a retry future

We target an OpenAI-compatible /v1/chat/completions endpoint. The retry block uses backoff::future::retry and converts our LlmError into transient or permanent backoff::Error variants.

use backoff::future::retry;
use serde::{Deserialize, Serialize};

#[derive(Serialize)]
struct ChatReq { model: String, messages: Vec<Message> }
#[derive(Serialize, Deserialize)]
struct Message { role: String, content: String }
#[derive(Deserialize)]
struct ChatResp { id: String, usage: Usage }
#[derive(Deserialize)]
struct Usage { total_tokens: u32 }

async fn chat(
    client: &reqwest::Client,
    url: &str,
    req: ChatReq,
) -> Result<ChatResp, LlmError> {
    let attempt = |_i: u32| async move {
        let resp = client.post(url).json(&req).send().await;
        match resp {
            Ok(r) if r.status().is_success() => {
                let json = r.json::<ChatResp>().await.map_err(LlmError::Transport)?;
                Ok(json)
            }
            Ok(r) => Err(backoff::Error::transient(classify(r.status(), None))),
            Err(e) if e.is_timeout() || e.is_connect() => {
                Err(backoff::Error::transient(LlmError::Transport(e)))
            }
            Err(e) => Err(backoff::Error::permanent(LlmError::Transport(e))),
        }
    };
    retry(backoff_policy(), attempt).await
}

Note the split: transport timeouts and connection resets are transient; TLS or DNS resolution failures are marked permanent because a local config bug won’t self-heal. For streaming responses, you need a different design—consume the stream and retry only if no bytes were received.

Step 5: Make retries safe with idempotency keys

Good rust llm api retry logic does not assume the server ignored a timed-out request. If your gateway honors a client-generated idempotency key, send one per logical call so a retry returns the original completion instead of double-generating.

use uuid::Uuid;

let idem = Uuid::new_v4().to_string();
client
    .post(url)
    .header("Idempotency-Key", idem)
    .json(&req)
    .send()
    .await

Without that support, only retry when the error occurred before the request reached the server (connection failed, timeout with no bytes sent). Our classification already treats those as transient. For mutations like fine-tune creation, never retry on ambiguity—surface the error.

Step 6: Mock the endpoint to prove retries fire

Stand up a tiny Python server that fails twice with 503 then succeeds. This validates the loop without spending tokens or hitting rate limits.

from flask import Flask, jsonify
app = Flask(__name__)
state = {"n": 0}

@app.route("/v1/chat/completions", methods=["POST"])
def completions():
    state["n"] += 1
    if state["n"] <= 2:
        return ("", 503)
    return jsonify({"id": "mock", "usage": {"total_tokens": 12}})

app.run(port=8080)

Point the Rust client at http://localhost:8080/v1/chat/completions. Run the binary; the server logs three POSTs, and the client should absorb the first two failures.

Step 7: Verify success and observe behavior

Success means the process exits zero, prints the mock response, and the server recorded exactly three hits. Wire a minimal main:

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = reqwest::Client::builder()
        .timeout(std::time::Duration::from_secs(20))
        .build()?;
    let req = ChatReq {
        model: "gpt-4o-mini".into(),
        messages: vec![Message { role: "user".into(), content: "hi".into() }],
    };
    let r = chat(&client, "http://localhost:8080/v1/chat/completions", req).await?;
    println!("got id {} tokens {}", r.id, r.usage.total_tokens);
    Ok(())
}

Run cargo run. If you see got id mock tokens 12, the rust llm api retry logic absorbed the 503s and recovered. For production, swap the URL to your real endpoint, add a tracing subscriber to count retry attempts per request, and alert if retry rates cross a threshold. You now have a client that retries only what should be retried, backs off cleanly, and can be tested offline.

Tagsrustretrieserror-handlingllm-client

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationError handling for LLM API clients in Rust with thiserrorA practical guide to building robust Rust LLM API clients with thiserror: modeling transport, provider, and parsing failures without leaking internals.API integrationRate limit handling in Rust LLM API clientsA practical guide to implementing robust Rust LLM API clients that gracefully handle rate limits using retries, backoff, and token buckets.API integrationCompiling a Rust LLM client to WASMBuild a rust LLM client that compiles to WebAssembly: set up toolchain, use reqwest wasm, bind with wasm-bindgen, and test in the browser.API integrationRust vs Go for building LLM API clientsA hands-on engineering comparison of Rust vs Go for building LLM API clients across latency, ergonomics, ecosystem, and cost tradeoffs with code examples and a verdict for different use cases.API integrationAsync LLM API calls in Rust with tokioA practical step-by-step guide to building a production-grade rust tokio async llm api client with concurrent calls, token streaming, retries, and a verification checklist.API integrationBuilding a Rust CLI tool for LLM chat completionsBuild a Rust CLI tool for LLM chat completions with the OpenAI API. Step-by-step tutorial covering reqwest, streaming, and model routing.

All rust llm api client posts →