n4nAI
Open console →
API integrationTutorialRust LLM API Client

Building a Rust CLI tool for LLM chat completions

Build a Rust CLI tool for LLM chat completions with the OpenAI API. Step-by-step tutorial covering reqwest, streaming, and model routing.

n4n Team2 min read461 words

Audio narration

Coming soon — every post will get a voice note here.

This tutorial builds a rust cli tool llm chat completions that talks to any OpenAI-compatible inference endpoint. You’ll get a working binary that sends a prompt, streams tokens to stdout, and respects provider routing hints—without pulling in a heavy SDK.

Prerequisites

  • Rust 1.70+ (rustc --version)
  • cargo
  • An API key from an OpenAI-compatible provider. If you want one endpoint for many models, n4n.ai provides an OpenAI-compatible base URL with automatic fallback across providers.
  • Basic comfort with async Rust and tokio.

Project scaffolding

cargo new rust-llm-cli
cd rust-llm-cli

Add the dependencies we actually need—no auto-generated client bloat:

[dependencies]
reqwest = { version = "0.12", features = ["json", "stream"] }
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
clap = { version = "4", features = ["derive"] }
anyhow = "1"
futures-util = "0.3"

CLI surface

Define arguments with clap. We keep it minimal: model, system prompt, user message, and base URL.

use clap::Parser;

#[derive(Parser, Debug)]
#[command(name = "rust-llm-cli")]
struct Args {
    /// Model ID, e.g. "gpt-4o-mini" or "anthropic/claude-3.5-sonnet"
    #[arg(short, long, default_value = "gpt-4o-mini")]
    model: String,

    /// System prompt
    #[arg(short, long, default_value = "You are a helpful assistant.")]
    system: String,

    /// User message
    #[arg(short, long)]
    message: String,

    /// Base URL for OpenAI-compatible API
    #[arg(long, default_value = "https://api.openai.com/v1")]
    base_url: String,
}

Non-streaming request first

Validate auth and JSON shape before adding streaming complexity. This call returns the full completion as one JSON object.

use anyhow::Result;
use reqwest::Client;
use serde_json::{json, Value};

async fn complete_once(args: &Args, api_key: &str) -> Result<Value> {
    let client = Client::new();
    let resp = client
        .post(format!("{}/chat/completions", args.base_url))
        .bearer_auth(api_key)
        .json(&json!({
            "model": args.model,
            "messages": [
                {"role": "system", "content": args.system},
                {"role": "user", "content": args.message}
            ],
            "stream": false
        }))
        .send()
        .await?
        .error_for_status()?
        .json::<Value>()
        .await?;
    Ok(resp)
}

Set OPENAI_API_KEY and run:

cargo run -- --message "What is 2+2?"

Expected truncated output (printed via dbg! or your own serializer):

{
  "id": "chatcmpl-abc123",
  "choices": [ { "message": { "content": "2 + 2 equals 4." } } ],
  "usage": { "total_tokens": 25 }
}

Streaming tokens

Production CLIs should stream. The endpoint returns text/event-stream with data: {json}\n\n lines. Parse with futures_util::StreamExt.

use futures_util::StreamExt;
use reqwest::header;

async fn stream_completion(args: &Args, api_key: &str) -> Result<()> {
    let client = Client::new();
    let resp = client
        .post(format!("{}/chat/completions", args.base_url))
        .bearer_auth(api_key)
        .header(header::ACCEPT, "text/event-stream")
        .json(&json!({
            "model": args.model,
            "messages": [
                {"role": "system", "content": args.system},
                {"role": "user", "content": args.message}
            ],
            "stream": true
        }))
        .send()
        .await?
        .error_for_status()?;

    let mut stream = resp.bytes_stream();
    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        let text = String::from_utf8_lossy(&chunk);
        for line in text.lines() {
            if let Some(data) = line.strip_prefix("data: ") {
                if data == "[DONE]" { return Ok(()); }
                if let Ok(v) = serde_json::from_str::<Value>(data) {
                    if let Some(tok) = v["choices"][0]["delta"]["content"].as_str() {
                        print!("{tok}");
                        std::io::Write::flush(&mut std::io::stdout())?;
                    }
                }
            }
        }
    }
    println!();
    Ok(())
}

The line-based loop handles most SSE frames. For strict correctness across chunk boundaries, buffer until newline, but for a rust cli tool llm chat completions this is robust enough.

Honor cache-control and routing

OpenAI-compatible gateways forward provider-specific headers. n4n.ai honors client routing directives and forwards provider cache-control hints; set them as headers on the request:

// example only, adjust to your gateway's actual header names
.headers({
    let mut h = header::HeaderMap::new();
    h.insert("X-Route", header::HeaderValue::from_static("fallback"));
    h
})

We don’t bake that into the default CLI, but the reqwest builder makes it a one-liner.

Main function

Wire env var, args, and stream.

#[tokio::main]
async fn main() -> Result<()> {
    let args = Args::parse();
    let api_key = std::env::var("OPENAI_API_KEY")
        .map_err(|_| anyhow::anyhow!("OPENAI_API_KEY not set"))?;
    stream_completion(&args, &api_key).await?;
    Ok(())
}

Build and run:

export OPENAI_API_KEY=sk-...
cargo run -- --model gpt-4o-mini --message "Explain Rust ownership in one sentence."

Expected streaming output (abridged):

Rust ownership ensures each value has a single owner, transferring or borrowing it safely to prevent data races and leaks.

Assembling the full binary

The complete src/main.rs for the rust cli tool llm chat completions looks like this:

use anyhow::Result;
use clap::Parser;
use futures_util::StreamExt;
use reqwest::header;
use serde_json::{json, Value};

#[derive(Parser, Debug)]
#[command(name = "rust-llm-cli")]
struct Args {
    #[arg(short, long, default_value = "gpt-4o-mini")]
    model: String,
    #[arg(short, long, default_value = "You are a helpful assistant.")]
    system: String,
    #[arg(short, long)]
    message: String,
    #[arg(long, default_value = "https://api.openai.com/v1")]
    base_url: String,
}

async fn stream_completion(args: &Args, api_key: &str) -> Result<()> {
    let client = Client::new();
    let resp = client
        .post(format!("{}/chat/completions", args.base_url))
        .bearer_auth(api_key)
        .header(header::ACCEPT, "text/event-stream")
        .json(&json!({
            "model": args.model,
            "messages": [
                {"role": "system", "content": args.system},
                {"role": "user", "content": args.message}
            ],
            "stream": true
        }))
        .send()
        .await?
        .error_for_status()?;

    let mut stream = resp.bytes_stream();
    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        let text = String::from_utf8_lossy(&chunk);
        for line in text.lines() {
            if let Some(data) = line.strip_prefix("data: ") {
                if data == "[DONE]" { return Ok(()); }
                if let Ok(v) = serde_json::from_str::<Value>(data) {
                    if let Some(tok) = v["choices"][0]["delta"]["content"].as_str() {
                        print!("{tok}");
                        std::io::Write::flush(&mut std::io::stdout())?;
                    }
                }
            }
        }
    }
    println!();
    Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
    let args = Args::parse();
    let api_key = std::env::var("OPENAI_API_KEY")
        .map_err(|_| anyhow::anyhow!("OPENAI_API_KEY not set"))?;
    stream_completion(&args, &api_key).await?;
    Ok(())
}

Testing without burning tokens

Spin up a local mock that emits SSE. This Python snippet is enough:

# mock_sse.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.end_headers()
        for i in ["Hello", " from", " mock"]:
            payload = json.dumps({"choices": [{"delta": {"content": i}}]})
            self.wfile.write(f"data: {payload}\n\n".encode())
        self.wfile.write(b"data: [DONE]\n\n")

HTTPServer(("127.0.0.1", 8080), H).serve_forever()

Run it, then point the CLI at the mock:

python mock_sse.py &
cargo run -- --base-url http://127.0.0.1:8080/v1 --message "test"

Output: Hello from mock

Error handling and exit codes

anyhow propagates errors; the process exits non-zero on failure. For a CLI, map rate-limit responses to a clear message:

if resp.status() == reqwest::StatusCode::TOO_MANY_REQUESTS {
    anyhow::bail!("Rate limited. Back off or switch model.");
}

Insert this before .error_for_status() or handle in a match. The user gets a non-zero exit and a readable string.

Shell integration

The current tool takes --message as an argument. To use it in pipelines, read from stdin when the flag is absent:

let message = if let Ok(m) = std::env::var("MESSAGE_FROM_STDIN") {
    m
} else {
    args.message.clone()
};

Then echo "summarize: $(cat file.txt)" | cargo run -- --message "$(cat)" works after a small wrapper. Keep the binary small; adding std::io::read_to_string is trivial.

Why direct reqwest instead of an SDK

The official OpenAI Rust crate is fine, but it hides the wire format and pulls in extra types. For a rust cli tool llm chat completions, direct reqwest keeps the dependency tree auditable and the binary under a few megabytes. You control retries, headers, and streaming parse logic.

Where to take it

Add --temperature, --max-tokens, or persist conversation history as JSONL. The core client is functional and speaks the dominant API shape. You now have a base to build agentic loops, RAG pipelines, or just a fast local chat wrapper.

Tagsrustclichat-completionstutorial

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationStreaming chat completions in Rust with SSEBuild rust sse streaming chat completions in Rust: configure tokio and reqwest, parse Server-Sent Events manually, extract deltas, and verify token output.API integrationThe async-openai crate: a quick start guideHands-on tutorial for the async-openai crate rust client: configure, call chat completions, stream tokens, and target any OpenAI-compatible LLM gateway.API integrationCalling the OpenAI API from Rust with reqwestA hands-on Rust tutorial for calling the OpenAI API with reqwest. Build a minimal chat client, handle streaming, errors, and auth step by step.API integrationCompiling a Rust LLM client to WASMBuild a rust LLM client that compiles to WebAssembly: set up toolchain, use reqwest wasm, bind with wasm-bindgen, and test in the browser.API integrationConnection pooling for LLM APIs with reqwestPractical guide to rust reqwest connection pooling for LLM APIs: configure clients, tune pool sizes, avoid pitfalls, and reuse connections efficiently.API integrationError handling for LLM API clients in Rust with thiserrorA practical guide to building robust Rust LLM API clients with thiserror: modeling transport, provider, and parsing failures without leaking internals.

All rust llm api client posts →