n4nAI
Open console →
API integrationHow-toRust LLM API Client

Async LLM API calls in Rust with tokio

A practical step-by-step guide to building a production-grade rust tokio async llm api client with concurrent calls, token streaming, retries, and a verification checklist.

n4n Team3 min read650 words

Audio narration

Coming soon — every post will get a voice note here.

Calling external LLM endpoints from Rust requires respecting async boundaries or you’ll starve the runtime. This tutorial builds a small but sturdy rust tokio async llm api client that sends chat completion requests concurrently, streams tokens, and survives provider errors. We’ll use tokio for the runtime, reqwest for HTTP, and serde for typing.

Step 1: Initialize the Cargo project and dependencies

Create a fresh binary crate:

cargo new llm_client && cd llm_client

Add the following to Cargo.toml. Pin major versions you trust; these are current as of tokio 1.x and reqwest 0.12.

[dependencies]
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["json", "stream"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
futures-util = "0.3"
anyhow = "1"
dotenvy = "0.15"

The stream feature on reqwest is required for token streaming. full tokio gives you macros, sync primitives, and time. Don’t enable full if you’re building a library—be precise about features to keep compile times sane.

Step 2: Define the API contract with serde

OpenAI-compatible chat endpoints expect a simple JSON shape. Model it explicitly; avoid serde_json::Value in hot paths because it defers errors to runtime.

use serde::{Deserialize, Serialize};

#[derive(Serialize, Debug, Clone)]
pub struct ChatMessage {
    pub role: String,
    pub content: String,
}

#[derive(Serialize, Debug, Clone)]
pub struct ChatRequest {
    pub model: String,
    pub messages: Vec<ChatMessage>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub stream: Option<bool>,
}

#[derive(Deserialize, Debug)]
pub struct ChatResponse {
    pub id: String,
    pub choices: Vec<Choice>,
    pub usage: Option<Usage>,
}

#[derive(Deserialize, Debug)]
pub struct Choice {
    pub message: ChatMessage,
}

#[derive(Deserialize, Debug)]
pub struct Usage {
    pub prompt_tokens: u32,
    pub completion_tokens: u32,
}

If you later swap providers, these structs stay stable because most gateways mirror the OpenAI schema. The clone derive on request types matters when you fan out.

Step 3: Write a basic async completion call

A single non-streaming call is the baseline. Use a shared reqwest::Client—it pools connections and is cheap to clone. Wrap it in a struct that is itself Clone.

use anyhow::Result;
use reqwest::Client;

#[derive(Clone)]
pub struct LlmClient {
    http: Client,
    api_key: String,
    base_url: String,
}

impl LlmClient {
    pub fn new(api_key: impl Into<String>, base_url: impl Into<String>) -> Self {
        Self {
            http: Client::new(),
            api_key: api_key.into(),
            base_url: base_url.into(),
        }
    }

    pub async fn complete(&self, req: &ChatRequest) -> Result<ChatResponse> {
        let resp = self
            .http
            .post(format!("{}/chat/completions", self.base_url))
            .bearer_auth(&self.api_key)
            .json(req)
            .send()
            .await?
            .error_for_status()?
            .json::<ChatResponse>()
            .await?;
        Ok(resp)
    }
}

Note the ? chain: error_for_status turns 4xx/5xx into errors before deserialization. That’s the correct order. Deserializing a 429 response into ChatResponse will panic or silently fail.

Step 4: Run multiple calls concurrently with tokio

Fan-out is where rust tokio async llm api code pays off. Spawning tasks or using buffer_unordered keeps the runtime saturated without blocking threads. Prefer buffer_unordered over tokio::spawn when you just need to collect results—it avoids task overhead and gives you backpressure.

use futures_util::StreamExt;

async fn batch_complete(client: &LlmClient, prompts: Vec<&str>) -> Result<Vec<ChatResponse>> {
    let futures = prompts.into_iter().map(|p| {
        let client = client.clone();
        let req = ChatRequest {
            model: "gpt-4o-mini".into(),
            messages: vec![ChatMessage { role: "user".into(), content: p.into() }],
            stream: None,
        };
        async move { client.complete(&req).await }
    });
    let results: Vec<Result<ChatResponse>> = futures::stream::iter(futures)
        .buffer_unordered(8) // cap concurrency
        .collect()
        .await;
    results.into_iter().collect()
}

buffer_unordered(8) issues up to eight simultaneous requests. Tune that number to your rate limit, not your hubris. If you need a global limit across many call sites, use a tokio::sync::Semaphore instead.

Step 5: Stream tokens with Server-Sent Events

For chat UIs you want tokens as they arrive. Set stream: Some(true) and parse the SSE byte stream. The response body is text/event-stream, not JSON.

use futures_util::StreamExt;
use reqwest::Response;

impl LlmClient {
    pub async fn stream_complete(&self, req: &ChatRequest) -> Result<()> {
        let mut req = req.clone();
        req.stream = Some(true);
        let resp: Response = self
            .http
            .post(format!("{}/chat/completions", self.base_url))
            .bearer_auth(&self.api_key)
            .json(&req)
            .send()
            .await?
            .error_for_status()?;

        let mut stream = resp.bytes_stream();
        while let Some(chunk) = stream.next().await {
            let chunk = chunk?;
            for line in std::str::from_utf8(&chunk)?.lines() {
                if let Some(data) = line.strip_prefix("data: ") {
                    if data == "[DONE]" { return Ok(()); }
                    if let Ok(v) = serde_json::from_str::<serde_json::Value>(data) {
                        if let Some(delta) = v["choices"][0]["delta"]["content"].as_str() {
                            print!("{delta}");
                        }
                    }
                }
            }
        }
        Ok(())
    }
}

This is deliberately minimal. Production code should handle retry_after headers, partial JSON across chunk boundaries, and client disconnects. SSE lines can be split across TCP segments; a robust client buffers until newline.

Step 6: Add retries and timeouts

Providers throttle. Wrap calls with a timeout and a bounded retry loop. Never retry on 4xx except 429 and 408.

use tokio::time::{sleep, Duration, timeout};

impl LlmClient {
    pub async fn complete_with_retry(&self, req: &ChatRequest) -> Result<ChatResponse> {
        let mut attempt = 0;
        loop {
            match timeout(Duration::from_secs(30), self.complete(req)).await {
                Ok(Ok(resp)) => return Ok(resp),
                Ok(Err(e)) if attempt < 3 => {
                    eprintln!("attempt {attempt} failed: {e}, retrying");
                    sleep(Duration::from_millis(200 * 2u64.pow(attempt))).await;
                    attempt += 1;
                }
                Ok(Err(e)) => return Err(e),
                Err(_) if attempt < 3 => {
                    attempt += 1;
                    continue;
                }
                Err(_) => return Err(anyhow::anyhow!("timeout after retries")),
            }
        }
    }
}

Exponential backoff with a cap prevents thundering herds. If you use a gateway, respect its Retry-After header instead of guessing.

Step 7: Route through a gateway

If you don’t want to hardcode a single vendor, point the base URL at an OpenAI-compatible gateway. n4n.ai exposes one endpoint that fronts 240+ models and automatically falls back when a provider is degraded, so the same LlmClient works without branching logic. You only change base_url and the model string.

let client = LlmClient::new(
    std::env::var("LLM_KEY").unwrap(),
    "https://api.n4n.ai/v1", // OpenAI-compatible
);

The client routing directives and cache-control headers pass through unchanged. This keeps your rust tokio async llm api client portable across providers.

Step 8: Verify the integration

Verification is concrete: run a small main that issues one streaming call and one batched call, then assert on output.

#[tokio::main]
async fn main() -> Result<()> {
    dotenvy::dotenv().ok();
    let key = std::env::var("OPENAI_API_KEY")?;
    let client = LlmClient::new(key, "https://api.openai.com/v1");

    let req = ChatRequest {
        model: "gpt-4o-mini".into(),
        messages: vec![ChatMessage { role: "user".into(), content: "Say hi".into() }],
        stream: None,
    };
    let resp = client.complete(&req).await?;
    assert!(!resp.choices.is_empty());
    println!("non-stream ok: {}", resp.choices[0].message.content);

    let prompts = vec!["1+1?", "2+2?"];
    let batch = batch_complete(&client, prompts).await?;
    assert_eq!(batch.len(), 2);
    println!("batch ok: {} responses", batch.len());

    client.stream_complete(&req).await?;
    println!("\nstream ok");
    Ok(())
}

Success criteria: the binary prints the completion text, the batch assertion holds, and cargo run exits 0. If you wired streaming, you’ll see tokens printed incrementally rather than all at once. For CI, replace the live call with a mocked HTTP server (e.g., wiremock) and assert on request shape.

Operational notes

A rust tokio async llm api client is only as good as its backpressure. Use Semaphore if you need global concurrency limits across many tasks, not just per batch. Meter token usage from the usage field; if you’re on a gateway that reports per-token metering, log it for cost tracking.

Don’t log full request bodies in production—prompts leak PII. Wrap Client with tracing instead of println!. Keep the reqwest::Client singleton; constructing a new one per call wastes connection pools and hurts latency.

That’s a complete, runnable path from cargo new to concurrent streaming calls with retries.

Tagsrusttokioasyncllm-client

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationCompiling a Rust LLM client to WASMBuild a rust LLM client that compiles to WebAssembly: set up toolchain, use reqwest wasm, bind with wasm-bindgen, and test in the browser.API integrationError handling for LLM API clients in Rust with thiserrorA practical guide to building robust Rust LLM API clients with thiserror: modeling transport, provider, and parsing failures without leaking internals.API integrationRate limit handling in Rust LLM API clientsA practical guide to implementing robust Rust LLM API clients that gracefully handle rate limits using retries, backoff, and token buckets.API integrationRetry logic for Rust LLM API clientsPractical guide to building robust rust llm api retry logic with exponential backoff, jitter, and correct error classification for LLM clients.API integrationRust vs Go for building LLM API clientsA hands-on engineering comparison of Rust vs Go for building LLM API clients across latency, ergonomics, ecosystem, and cost tradeoffs with code examples and a verdict for different use cases.API integrationBuilding a Rust CLI tool for LLM chat completionsBuild a Rust CLI tool for LLM chat completions with the OpenAI API. Step-by-step tutorial covering reqwest, streaming, and model routing.

All rust llm api client posts →