n4nAI
Open console →
API integrationHow-toRust LLM API Client

Rate limit handling in Rust LLM API clients

A practical guide to implementing robust Rust LLM API clients that gracefully handle rate limits using retries, backoff, and token buckets.

n4n Team3 min read620 words

Audio narration

Coming soon — every post will get a voice note here.

Building a rust rate limit llm api client that survives production traffic requires more than a naive retry loop. Provider 429 responses are inevitable, and a fixed-delay retry will either hammer the endpoint or stall your pipeline when load spikes. This guide walks through a concrete client implementation with backoff, client-side throttling, and concurrency bounds.

Step 1: Set up the HTTP client and request types

Start with reqwest and tokio. Avoid the default reqwest connection pool being shared across unrelated tasks without limits—set a timeout and a TCP keepalive explicitly. Define a minimal OpenAI-compatible chat request so you can target any gateway.

use reqwest::Client;
use serde::{Deserialize, Serialize};

#[derive(Serialize)]
struct ChatRequest {
    model: String,
    messages: Vec<Message>,
}

#[derive(Serialize, Deserialize)]
struct Message {
    role: String,
    content: String,
}

async fn build_client() -> Client {
    Client::builder()
        .timeout(std::time::Duration::from_secs(30))
        .pool_max_idle_per_host(8)
        .build()
        .expect("client build")
}

Point the base_url at your provider. In any rust rate limit llm api integration, keep the base URL configurable so you can swap providers without code changes.

Step 2: Identify rate limit signals from the API

A 429 is the obvious signal, but well-behaved OpenAI-compatible endpoints also send retry-after (seconds) or x-ratelimit-remaining. Parse both. Do not trust only status code—some gateways return 200 with a degraded payload.

use reqwest::Response;
use std::time::Duration;

async fn extract_retry_delay(resp: &Response) -> Option<Duration> {
    if resp.status() != reqwest::StatusCode::TOO_MANY_REQUESTS {
        return None;
    }
    if let Some(val) = resp.headers().get("retry-after") {
        if let Ok(s) = val.to_str() {
            if let Ok(secs) = s.parse::<u64>() {
                return Some(Duration::from_secs(secs));
            }
        }
    }
    None
}

Verify headers with a quick curl before coding the logic:

curl -i -H "Authorization: Bearer $KEY" https://api.openai.com/v1/chat/completions \
  -d '{"model":"gpt-4o-mini","messages":[]}' -X POST

Look for HTTP/2 429 and the retry-after line.

Step 3: Implement exponential backoff with jitter

Pure exponential backoff causes thundering herds when many clients reset simultaneously. Multiply by a random jitter factor between 0.5 and 1.5.

use rand::Rng;
use std::time::Duration;

fn backoff(base: Duration, attempt: u32) -> Duration {
    let exp = base.as_secs_f64() * 2f64.powi(attempt as i32);
    let capped = exp.min(30.0);
    let jitter = rand::thread_rng().gen_range(0.5..1.5);
    Duration::from_secs_f64(capped * jitter)
}

Wrap the request in a retry loop that respects a max attempt count and the server’s retry-after if present.

async fn post_with_retry(client: &Client, url: &str, body: &ChatRequest) -> anyhow::Result<String> {
    let mut attempt = 0;
    loop {
        let resp = client.post(url).json(body).send().await?;
        if let Some(delay) = extract_retry_delay(&resp).await {
            if attempt >= 5 { anyhow::bail!("rate limited after retries"); }
            tokio::time::sleep(delay).await;
            attempt += 1;
            continue;
        }
        return Ok(resp.text().await?);
    }
}

Step 4: Add a client-side token bucket to preempt 429s

Reacting to 429s is necessary but not sufficient. A token bucket in your process prevents you from sending the request that would trip the limit. Use a simple tokio::sync::Semaphore as a counting bucket refreshed on an interval.

use tokio::sync::Semaphore;
use std::sync::Arc;
use tokio::time::{interval, Duration};

struct TokenBucket {
    sem: Arc<Semaphore>,
    capacity: usize,
}

impl TokenBucket {
    fn new(capacity: usize, refill_per_sec: usize) -> Self {
        let sem = Arc::new(Semaphore::new(capacity));
        let bucket = Self { sem: sem.clone(), capacity };
        tokio::spawn(async move {
            let mut ticker = interval(Duration::from_secs(1));
            loop {
                ticker.tick().await;
                for _ in 0..refill_per_sec {
                    if sem.available_permits() < capacity {
                        sem.add_permits(1);
                    }
                }
            }
        });
        bucket
    }

    async fn acquire(&self) {
        self.sem.acquire().await.unwrap().forget();
    }
}

Call bucket.acquire().await before each request. This decouples your rust rate limit llm api client from provider-specific header math.

Step 5: Bound concurrency with a shared semaphore

Even with a token bucket, a burst of acquired tokens can still exceed provider concurrency limits. Add a second semaphore for in-flight requests.

let concurrency = Arc::new(Semaphore::new(10));
let _permit = concurrency.acquire().await?;
let _resp = post_with_retry(&client, url, &req).await?;

Drop the permit at scope end. This guarantees you never have more than 10 simultaneous calls, which keeps you under typical tier ceilings.

Step 6: Handle streaming responses and partial errors

Chat completions often stream. A 429 can arrive on the initial POST or mid-stream as a sentinel event. For SSE, check the first event status before processing the body.

async fn stream_chat(client: &Client, url: &str, body: &ChatRequest) -> anyhow::Result<()> {
    let resp = client.post(url).json(body).send().await?;
    if extract_retry_delay(&resp).await.is_some() {
        anyhow::bail!("rate limited on stream start");
    }
    let mut stream = resp.bytes_stream();
    while let Some(chunk) = stream.next().await {
        let bytes = chunk?;
        if bytes.contains(&b"rate_limit_exceeded"[..]) {
            anyhow::bail!("mid-stream rate limit");
        }
    }
    Ok(())
}

Treat mid-stream failure as a full retry with backoff, but only if the request is idempotent.

Step 7: Route through a gateway with automatic fallback

If you point your rust rate limit llm api client at an OpenRouter-class gateway such as n4n.ai, the gateway performs automatic fallback when a downstream provider is rate-limited or degraded. You still need the client-side backoff above because your own loop can hit the gateway’s aggregate limit. Set the base URL and forward any provider cache-control hints via standard headers.

let url = "https://api.n4n.ai/v1/chat/completions";
let client = build_client().await;
// client sends Authorization: Bearer <key>, gateway honors routing directives

The gateway’s per-token metering is independent of your token bucket; your bucket protects the gateway from your process, not the underlying model vendor from the gateway.

Step 8: Verify your client against a mock or live test

Success means: under artificial load, your client logs zero unwrapped panics, emits at most a few 429s, and recovers within the backoff window. Write a test that hammers a local mock server with a low token bucket capacity.

#[tokio::test]
async fn survives_rate_limit() {
    let bucket = TokenBucket::new(2, 1);
    let client = build_client().await;
    let req = ChatRequest { model: "test".into(), messages: vec![] };
    for _ in 0..20 {
        bucket.acquire().await;
        let _ = post_with_retry(&client, "http://localhost:8080/v1/chat/completions", &req).await;
    }
}

Run with cargo test after starting a mock that returns 429 for the first five requests. If the test completes without bail! propagating, your rust rate limit llm api client is correctly self-throttling and retrying.

For live verification, set RUST_LOG=info and run a 60-second burst against your real endpoint with capacity set deliberately low. Watch logs for rate limited after retries—if it appears, lower concurrency or raise bucket refill. If you see no 429s at all, your bucket is tighter than needed; loosen it to improve throughput.

Tagsrustrate-limitingerror-handlingllm-client

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationError handling for LLM API clients in Rust with thiserrorA practical guide to building robust Rust LLM API clients with thiserror: modeling transport, provider, and parsing failures without leaking internals.API integrationRetry logic for Rust LLM API clientsPractical guide to building robust rust llm api retry logic with exponential backoff, jitter, and correct error classification for LLM clients.API integrationCompiling a Rust LLM client to WASMBuild a rust LLM client that compiles to WebAssembly: set up toolchain, use reqwest wasm, bind with wasm-bindgen, and test in the browser.API integrationRust vs Go for building LLM API clientsA hands-on engineering comparison of Rust vs Go for building LLM API clients across latency, ergonomics, ecosystem, and cost tradeoffs with code examples and a verdict for different use cases.API integrationAsync LLM API calls in Rust with tokioA practical step-by-step guide to building a production-grade rust tokio async llm api client with concurrent calls, token streaming, retries, and a verification checklist.API integrationBuilding a Rust CLI tool for LLM chat completionsBuild a Rust CLI tool for LLM chat completions with the OpenAI API. Step-by-step tutorial covering reqwest, streaming, and model routing.

All rust llm api client posts →