Parsing LLM API responses in Rust with serde_json

Calling an LLM API from Rust is easy with reqwest, but turning the JSON payload into typed structs is where subtle bugs creep in. This tutorial builds a small client that parses OpenAI-compatible chat completions using rust serde_json llm api patterns you can reuse across providers. We’ll go from static parsing to a live async request, with checkpoints showing exact output.

Prerequisites

You need a working Rust toolchain (1.75+, edition 2021) and cargo. We’ll use these crates:

[dependencies]
reqwest = { version = "0.12", features = ["json", "rustls-tls"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
tokio = { version = "1", features = ["full"] }
anyhow = "1.0"

Set your API key in the environment:

export OPENAI_API_KEY="sk-..."

The core of any rust serde_json llm api integration is a typed response model. Skip serde_json::Value in production—you want compile-time checks on field names and types.

Define the response shape

OpenAI-compatible chat completions share a predictable schema. Map it to Rust structs with serde::Deserialize:

use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct ChatCompletion {
    id: String,
    #[serde(rename = "object")]
    kind: String,
    created: u64,
    model: String,
    choices: Vec<Choice>,
    usage: Usage,
    #[serde(default)]
    system_fingerprint: Option<String>,
}

#[derive(Debug, Deserialize)]
struct Choice {
    index: u32,
    message: Message,
    #[serde(rename = "finish_reason")]
    finish_reason: Option<String>,
}

#[derive(Debug, Deserialize)]
struct Message {
    role: String,
    content: String,
}

#[derive(Debug, Deserialize)]
struct Usage {
    prompt_tokens: u32,
    completion_tokens: u32,
    total_tokens: u32,
}

Note the #[serde(rename)] attributes: the wire format uses snake_case, but Rust idioms prefer clear names. system_fingerprint is optional and not present on every provider, so we mark it #[serde(default)].

Parse a static payload first

Before hitting the network, verify the mapping against a known JSON blob:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1699000000,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello, world." },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 3,
    "total_tokens": 13
  }
}

Drop it into a minimal binary:

fn main() -> anyhow::Result<()> {
    let data = r#"{
      "id": "chatcmpl-123",
      "object": "chat.completion",
      "created": 1699000000,
      "model": "gpt-4o-mini",
      "choices": [
        {
          "index": 0,
          "message": { "role": "assistant", "content": "Hello, world." },
          "finish_reason": "stop"
        }
      ],
      "usage": {
        "prompt_tokens": 10,
        "completion_tokens": 3,
        "total_tokens": 13
      }
    }"#;

    let resp: ChatCompletion = serde_json::from_str(data)?;
    println!("Model: {}", resp.model);
    println!("Content: {}", resp.choices[0].message.content);
    println!("Total tokens: {}", resp.usage.total_tokens);
    Ok(())
}

Expected output:

Model: gpt-4o-mini
Content: Hello, world.
Total tokens: 13

If you see a missing field error, your struct is stricter than the payload. That’s good—you’ve caught a contract mismatch early.

Make a real async request

Now replace the static string with a live call. We’ll post to an OpenAI-compatible endpoint and deserialize the response stream directly.

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = reqwest::Client::new();
    let body = serde_json::json!({
        "model": "gpt-4o-mini",
        "messages": [{"role": "user", "content": "Say hello."}]
    });

    let resp = client
        .post("https://api.openai.com/v1/chat/completions")
        .bearer_auth(std::env::var("OPENAI_API_KEY")?)
        .json(&body)
        .send()
        .await?
        .json::<ChatCompletion>()
        .await?;

    println!("Model: {}", resp.model);
    println!("Reply: {}", resp.choices[0].message.content);
    println!("Tokens: {}", resp.usage.total_tokens);
    Ok(())
}

If you point this at n4n.ai’s single OpenAI-compatible endpoint, the same structs work across 240+ models because the gateway normalizes the response shape and honors provider cache-control hints—no client-side branching needed.

Run it:

cargo run --release

Typical output:

Model: gpt-4o-mini
Reply: Hello! How can I help you today?
Tokens: 17

Handle optional and null content

Real responses are messier. Some models return null content when they emit tool calls, or omit finish_reason on streamed final chunks. Make Message resilient:

#[derive(Debug, Deserialize)]
struct Message {
    role: String,
    #[serde(default)]
    content: Option<String>,
    #[serde(default)]
    tool_calls: Option<Vec<serde_json::Value>>,
}

Now extract safely:

let content = resp.choices[0].message.content.clone().unwrap_or_default();
if content.is_empty() {
    println!("Model returned no text; likely tool calls.");
} else {
    println!("Reply: {content}");
}

I prefer Option<String> over String with #[serde(default)] because it forces the caller to handle the absent case. Silent empty strings hide bugs.

Deal with provider extensions

Providers tack on fields like logprobs, system_fingerprint, or vendor-specific metadata. Serde ignores unknown fields by default, which is usually what you want. If you need strictness during tests, add #[serde(deny_unknown_fields)] to a test-only copy of the struct to catch drift.

When extending the rust serde_json llm api client, keep the base struct minimal and put extras in a separate map:

#[derive(Debug, Deserialize)]
struct ChatCompletion {
    // ... core fields ...
    #[serde(flatten)]
    extra: std::collections::HashMap<String, serde_json::Value>,
}

flatten collects unknown keys without losing them. Avoid this in hot paths—it allocates—but for config-time parsing it’s fine.

Reusable module layout

Package the parser as its own module so binaries stay thin:

// src/llm.rs
use serde::Deserialize;

#[derive(Debug, Deserialize)]
pub struct ChatCompletion { /* ... */ }

#[derive(Debug, Deserialize)]
pub struct Choice { /* ... */ }

#[derive(Debug, Deserialize)]
pub struct Message { /* ... */ }

#[derive(Debug, Deserialize)]
pub struct Usage { /* ... */ }

pub fn parse(data: &str) -> anyhow::Result<ChatCompletion> {
    Ok(serde_json::from_str(data)?)
}

Then in main.rs:

mod llm;
use llm::ChatCompletion;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // ... reqwest call ...
    let completion: ChatCompletion = resp.json().await?;
    Ok(())
}

This separation makes it trivial to swap HTTP clients or add middleware without touching the parsing logic.

Closing notes

Typed parsing with serde_json turns a loosely specified JSON API into a compile-checked surface. Start strict, add Option and #[serde(default)] only where the contract is genuinely optional, and keep provider-specific fields out of your core types. The rust serde_json llm api approach above scales from a weekend script to a gateway client handling dozens of models.