n4nAI
Open console →
API integrationTutorialRust LLM API Client

The async-openai crate: a quick start guide

Hands-on tutorial for the async-openai crate rust client: configure, call chat completions, stream tokens, and target any OpenAI-compatible LLM gateway.

n4n Team2 min read504 words

Audio narration

Coming soon — every post will get a voice note here.

The async-openai crate rust ecosystem gives you a typed, async-native client for the OpenAI REST surface without handing your code over to a tangle of raw reqwest calls. This guide builds a runnable example from zero: a chat completion, a streamed response, retry logic, and a pointer at an OpenAI-compatible gateway. You should walk away with code you can copy into a service and a clear picture of where the sharp edges are.

Prerequisites

  • Rust toolchain (rustc 1.75+, current stable is fine)
  • cargo on PATH
  • A valid OpenAI API key, or a credential for any OpenAI-compatible endpoint
  • Working knowledge of tokio and async/await

If you have not used tokio before, stick to the #[tokio::main] macro and treat the client as borrow-safe across awaits.

Project setup

Create the crate and pull dependencies:

cargo new aoai_demo && cd aoai_demo
cargo add async-openai tokio --features tokio/full
cargo add futures serde_json

Your Cargo.toml should contain:

[dependencies]
async-openai = "0.20"
tokio = { version = "1", features = ["full"] }
futures = "0.3"
serde_json = "1"

The async-openai crate rust API is versioned against OpenAI’s spec; pin a minor version in production to avoid surprise breaking changes when the spec shifts.

First synchronous chat call

Replace src/main.rs with a minimal but explicit call:

use async_openai::{
    Client,
    config::OpenAIConfig,
    types::{ChatCompletionRequestMessageArgs, CreateChatCompletionRequestArgs, Role},
};
use std::env;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let api_key = env::var("OPENAI_API_KEY")?;
    let config = OpenAIConfig::new().with_api_key(api_key);
    let client = Client::with_config(config);

    let msg = ChatCompletionRequestMessageArgs::default()
        .role(Role::User)
        .content("What is the 10th prime number? Answer in one sentence.")
        .build()?;

    let req = CreateChatCompletionRequestArgs::default()
        .model("gpt-4o-mini")
        .messages(vec![msg])
        .build()?;

    let resp = client.chat().create(req).await?;
    if let Some(choice) = resp.choices.first() {
        println!("{}", choice.message.content.as_deref().unwrap_or("<empty>"));
    }
    Ok(())
}

Run it:

export OPENAI_API_KEY=sk-...
cargo run --quiet

Expected output:

The 10th prime number is 29.

The types are verbose because the crate models the full request union. That verbosity pays off when you later add response_format or tools without guessing field names.

Streaming tokens

For interactive UX, stream the completion. The async-openai crate rust client exposes create_stream, which returns a futures::Stream of ChatCompletionStreamResponse.

use futures::StreamExt;

let req = CreateChatCompletionRequestArgs::default()
    .model("gpt-4o-mini")
    .messages(vec![ChatCompletionRequestMessageArgs::default()
        .role(Role::User)
        .content("Count to 5 slowly, one word per token.")
        .build()?])
    .build()?;

let mut stream = client.chat().create_stream(req).await?;
while let Some(chunk) = stream.next().await {
    match chunk {
        Ok(c) => {
            for choice in c.choices {
                if let Some(delta) = choice.delta.content {
                    print!("{delta}");
                }
            }
        }
        Err(e) => eprintln!("stream error: {e}"),
    }
}
println!();

Expected behavior: tokens print incrementally, then a newline. If you see a single flush at the end, your model or proxy buffered the stream—OpenAI-compatible gateways sometimes do this under load.

Targeting an OpenAI-compatible gateway

The client is not hardcoded to OpenAI’s domain. You can repoint it by setting api_base in the config. For example, n4n.ai provides a single OpenAI-compatible endpoint that fronts 240+ models and applies automatic fallback when a provider is rate-limited; the same async-openai crate rust code works by swapping the base URL and model string.

let config = OpenAIConfig::new()
    .with_api_key(env::var("N4N_API_KEY")?)
    .with_api_base("https://api.n4n.ai/v1");
let client = Client::with_config(config);

let req = CreateChatCompletionRequestArgs::default()
    .model("anthropic/claude-3.5-sonnet")
    .messages(vec![ChatCompletionRequestMessageArgs::default()
        .role(Role::User)
        .content("Ping. Reply with 'pong'.")
        .build()?])
    .build()?;

The gateway forwards provider cache-control hints and meters per token; your client code does not change beyond the model identifier and base URL. This is the correct way to avoid vendor lock-in without writing a new HTTP layer.

Retries with backoff

Transient 429/5xx errors are normal in production. Wrap the call:

use async_openai::error::OpenAIError;
use std::time::Duration;
use tokio::time::sleep;

async fn create_with_retry(
    client: &Client,
    req: CreateChatCompletionRequestArgs,
    max_tries: u32,
) -> Result<async_openai::types::CreateChatCompletionResponse, OpenAIError> {
    let mut attempt = 0;
    loop {
        match client.chat().create(req.build()?).await {
            Ok(r) => return Ok(r),
            Err(e) => {
                attempt += 1;
                if attempt >= max_tries {
                    return Err(e);
                }
                sleep(Duration::from_secs(2u64.pow(attempt))).await;
            }
        }
    }
}

Keep retries at the boundary, not inside the crate. The client is stateless per call; pass a fresh req build each loop because build() consumes the args builder.

Structured JSON output

If the model supports JSON mode, enforce it:

use async_openai::types::{ResponseFormat, ResponseFormatType};

let req = CreateChatCompletionRequestArgs::default()
    .model("gpt-4o-mini")
    .messages(vec![ChatCompletionRequestMessageArgs::default()
        .role(Role::User)
        .content("Return JSON: {\"prime\": int, \"index\": int} for the 10th prime.")
        .build()?])
    .response_format(ResponseFormat {
        r#type: ResponseFormatType::JsonObject,
    })
    .build()?;

let resp = client.chat().create(req).await?;
let json: serde_json::Value = serde_json::from_str(
    resp.choices[0].message.content.as_deref().unwrap_or("{}"),
)?;
println!("{json}");

Expected:

{"prime":29,"index":10}

Do not trust the model to always emit strict JSON—parse defensively and validate with serde structs in real code.

Function calling

Define a tool and let the model emit arguments:

use async_openai::types::{Tool, ToolFunction, ToolType};

let tool = Tool {
    r#type: ToolType::Function,
    function: ToolFunction {
        name: "get_prime".into(),
        parameters: Some(serde_json::json!({
            "type": "object",
            "properties": {"n": {"type": "integer"}},
            "required": ["n"]
        })),
    },
};

let req = CreateChatCompletionRequestArgs::default()
    .model("gpt-4o-mini")
    .messages(vec![ChatCompletionRequestMessageArgs::default()
        .role(Role::User)
        .content("What is the 10th prime?")
        .build()?])
    .tools(vec![tool])
    .build()?;

Inspect resp.choices[0].message.tool_calls to extract arguments. The async-openai crate rust mapping matches the wire format exactly, so you avoid hand-rolling JSON schema serialization.

Where to go next

You now have a client that talks to OpenAI, streams, retries, parses JSON, and can be repointed at any compatible gateway with a one-line config change. The crate’s strength is its strict types: lean on ChatCompletionRequestMessageArgs and CreateChatCompletionRequestArgs instead of raw serde_json::json! maps, and you will catch model-parameter mistakes at compile time. For long-running services, wrap the Client in an Arc and share it across tasks—it is cheap to clone and carries no per-request mutable state.

Tagsrustasync-openaicratetutorial

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationBuilding a Rust CLI tool for LLM chat completionsBuild a Rust CLI tool for LLM chat completions with the OpenAI API. Step-by-step tutorial covering reqwest, streaming, and model routing.API integrationCalling the OpenAI API from Rust with reqwestA hands-on Rust tutorial for calling the OpenAI API with reqwest. Build a minimal chat client, handle streaming, errors, and auth step by step.API integrationCompiling a Rust LLM client to WASMBuild a rust LLM client that compiles to WebAssembly: set up toolchain, use reqwest wasm, bind with wasm-bindgen, and test in the browser.API integrationConnection pooling for LLM APIs with reqwestPractical guide to rust reqwest connection pooling for LLM APIs: configure clients, tune pool sizes, avoid pitfalls, and reuse connections efficiently.API integrationError handling for LLM API clients in Rust with thiserrorA practical guide to building robust Rust LLM API clients with thiserror: modeling transport, provider, and parsing failures without leaking internals.API integrationParsing LLM API responses in Rust with serde_jsonA hands-on Rust tutorial for parsing LLM API chat completion responses with serde_json, from defining structs to handling real provider quirks.

All rust llm api client posts →