Request and response shapes in chat completions

The chat completions request response shape is the JSON schema that governs how a client sends conversation state and sampling parameters to an LLM endpoint and how that endpoint returns generated messages, finish reasons, and token accounting. This chat completions request response shape is the lingua franca of OpenAI-compatible gateways: every field from messages[].role to usage.completion_tokens drives routing, caching, and billing decisions downstream. If you treat it as a black box, you will mis-handle streaming, drop tool calls, or double-count tokens.

Request shape: the input side

A minimal request posts to /v1/chat/completions with a JSON body. The three required keys are model and messages; stream is optional but changes the response contract entirely. Everything else tunes behavior.

{
  "model": "gpt-4o-mini",
  "messages": [
    {"role": "system", "content": "You are a terse API reviewer."},
    {"role": "user", "content": "Explain idempotency keys."}
  ],
  "temperature": 0.2,
  "max_tokens": 512
}

messages is an ordered array. Each item has role (system, user, assistant, or tool) and content (string or structured parts). The system message sets behavior; many models ignore it if placed after the first user turn, so inject it first.

Parameters like temperature, top_p, max_tokens, stop, presence_penalty are hints. A gateway may forward them or clamp them per model limits. If you send max_tokens larger than the model context minus prompt, the provider errors or truncates.

Routing and cache hints

The request can carry extension fields. OpenAI-compatible gateways accept stream_options: {"include_usage": true} to force usage in the streaming final chunk. Some accept user for abuse tracking. n4n.ai honors client routing directives via headers like x-routing-prefer and forwards provider cache-control hints such as cache_control on system content, so the chat completions request response shape directly affects cache hit rate.

{
  "model": "claude-3-5-sonnet",
  "messages": [
    {"role": "system", "content": "Long static spec", "cache_control": {"type": "ephemeral"}}
  ]
}

Reordering or mutating that system string defeats prefix caching and forces a full recompute upstream.

Response shape: non-streaming

A synchronous call returns one JSON object. The top level has id, object, created, model, choices, usage.

{
  "id": "chatcmpl-8x",
  "object": "chat.completion",
  "created": 1690000000,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "Idempotency keys prevent duplicate side effects..."},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 24, "completion_tokens": 58, "total_tokens": 82}
}

choices is an array to support n > 1. Each message mirrors the request role scheme. finish_reason tells you why generation stopped: stop, length, tool_calls, content_filter. Ignore it and you may ship truncated text to a user.

usage is mandatory in non-streaming. It is the only authoritative token count. Do not estimate from string length; tokenizer mismatch will break your metering.

Streaming shape: delta chunks

Set "stream": true and you get text/event-stream chunks. Each chunk is a chat.completion.chunk object with choices[].delta containing incremental fields.

{"id":"chatcmpl-9","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}
{"id":"chatcmpl-9","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Idem"},"finish_reason":null}]}
{"id":"chatcmpl-9","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"potency"},"finish_reason":null}]}
{"id":"chatcmpl-9","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

The first chunk often carries role. Subsequent chunks carry content fragments. The final chunk has empty delta and a finish_reason. Usage appears only if you sent stream_options: {"include_usage": true}; then a final chunk with "delta": {} and a usage object arrives after finish_reason.

Concatenating delta.content yields the full text. Do not assume usage is present in every chunk; clients that sum tokens from deltas will undercount.

Error response shape

A non-2xx returns a distinct envelope. The chat completions request response shape includes this error contract:

{
  "error": {
    "message": "This model's maximum context length is 8192 tokens.",
    "type": "invalid_request_error",
    "code": "context_length_exceeded"
  }
}

Clients must check HTTP status and parse error, not assume choices exists. A 200 with a malformed body is not part of the spec; gateways that provide automatic fallback rely on these error codes to switch providers.

Why the chat completions request response shape matters

Get the shape wrong and you break three production concerns:

1. Routing – Gateways inspect model and headers to pick a provider. A malformed messages array triggers a 400 before any model runs.
2. Caching – Provider-side prefix caching keys on exact system content and cache_control markers. Reordering messages invalidates the cache.
3. Metering – Per-token billing depends on the usage object. If you drop streaming usage, you cannot reconcile cost.

A gateway such as n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and applies automatic fallback when a provider is rate-limited. It parses your chat completions request response shape to meter per-token usage and to forward cache hints; a malformed shape defeats fallback logic.

Concrete example: minimal Python client

Below is a correct non-streaming call using requests. No SDK magic.

import requests, os

resp = requests.post(
    "https://api.openai.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},
    json={
        "model": "gpt-4o-mini",
        "messages": [
            {"role": "system", "content": "Answer in JSON."},
            {"role": "user", "content": "List two HTTP methods."}
        ],
        "temperature": 0
    },
    timeout=30
)
data = resp.json()
print(data["choices"][0]["message"]["content"])
print(data["usage"]["total_tokens"])

Swap the base URL and key to target a gateway. The shape is identical.

For streaming with usage:

import requests, json

with requests.post(url, headers=hdr, json={**payload, "stream": True,
        "stream_options": {"include_usage": True}}, stream=True) as r:
    for line in r.iter_lines():
        if not line or not line.startswith(b"data: "):
            continue
        chunk = json.loads(line[6:])
        if "usage" in chunk:
            print("usage", chunk["usage"])

Common misconceptions

“System messages are optional”

For instruction-tuned models, omitting system defaults to generic behavior. But some gateways and models treat the first user message as system. If you rely on consistent behavior, always send an explicit system role first.

“Streaming chunks contain running usage”

They do not. Unless you explicitly request include_usage, the usage field is absent from all streamed chunks. Engineers who approximate cost by counting words inflate or deflate bills by 2–5x depending on the tokenizer.

“Tool calls arrive as text”

When a model emits tool_calls, the assistant message contains a tool_calls array with function.name and function.arguments (a JSON string). Treating it as content drops the call. The chat completions request response shape separates these; respect it.

{
  "role": "assistant",
  "tool_calls": [
    {"id": "call_1", "type": "function",
     "function": {"name": "get_weather", "arguments": "{\"city\":\"SF\"}"}}
  ]
}

“The same request always returns the same id”

id is provider-generated per call. It is not a dedupe key. Use your own user or correlation id.

“max_tokens limits total context”

It limits completion length only. Prompt length is bounded by model context window minus max_tokens. Send a 30k token prompt to an 8k model and you get a 400 before generation.

“n>1 just repeats the answer”

Setting n: 3 returns three independent completions in choices. Each carries its own finish_reason and contributes to completion_tokens multiplicatively. Your metering code must loop, not index [0].

Multimodal and structured extensions

Vision models accept content as an array of {"type":"image_url","image_url":{"url":"data:..."}} alongside text. The chat completions request response shape absorbs this without a new endpoint. Structured outputs add response_format: {"type":"json_schema","json_schema":{...}}; the response content is still a string conforming to schema.

{
  "model": "gpt-4o",
  "messages": [
    {"role":"user","content":[
      {"type":"text","text":"Describe image"},
      {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}
    ]}
  ]
}

If you request logprobs: true, the non-streaming response adds choices[].logprobs with tokens and token_logprobs arrays. The shape stays consistent; you just descend one level deeper.

Checklist before you ship

• Validate messages order: system first, then alternating user/assistant.
• Set stream_options.include_usage if you bill on streamed calls.
• Parse finish_reason and handle length by truncating UI, not by retrying blindly.
• Forward cache_control on static prefixes to exploit provider caching.
• Treat usage as authoritative; never sum delta.content lengths.
• Use tool_calls fields, not regex on content.
• Check error envelope on non-2xx; do not assume choices exists.
• Loop over choices when n > 1.

The chat completions request response shape is small but unforgiving. Learn its edges and your gateway integration stays boring—which is the goal.