How GPT-5 handles text, image, and audio in one model

GPT-5 multimodal capabilities unify text, image, and audio processing in a single model endpoint, eliminating the orchestration layer most teams currently maintain. Instead of stitching together Whisper, CLIP, and a language model, you send one request and get one response. This changes how you design pipelines, handle fallbacks, and budget tokens. The guide below walks through the request format, streaming behavior, cost dynamics, and the failure modes you will hit in production.

Request format and content blocks

The API accepts a single messages array where each message can contain multiple content blocks. Each block has a type field — text, image_url, or input_audio — and the model processes them in order. This is not three separate endpoints; it is one conversation history with mixed modalities.

{
  "model": "gpt-5",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Transcribe and summarize this meeting recording." },
        {
          "type": "input_audio",
          "input_audio": {
            "data": "base64_encoded_audio",
            "format": "wav"
          }
        }
      ]
    }
  ],
  "modalities": ["text", "audio"],
  "audio": { "voice": "alloy", "format": "wav" }
}

Key fields to understand:

• modalities declares what you want back. Omit audio if you only need text. Requesting audio when you don’t need it burns output tokens and latency.
• audio.format controls the response encoding. wav is uncompressed and largest; mp3 and opus are smaller but add decode overhead on the client.
• input_audio.format must match the actual encoding. The model does not auto-detect; mismatched formats return garbage transcriptions.

Images follow the same pattern. You can pass a data URI or an HTTPS URL. The model downloads URLs at request time, so latency depends on the image host.

{
  "type": "image_url",
  "image_url": {
    "url": "https://cdn.example.com/chart.png",
    "detail": "high"
  }
}

The detail parameter (low, high, auto) controls the vision encoder’s tile count. high uses more tokens and catches fine text; low downsamples aggressively. For screenshots with UI text, use high. For photos where you only need scene understanding, low saves 60-70% of image tokens.

Streaming and partial results

GPT-5 streams text tokens as usual. Audio streams as base64-encoded chunks in the audio field of each delta. You cannot stream images — the model either generates an image (via tool call to an image model) or it doesn’t. There is no progressive image decode.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def stream_multimodal():
    stream = await client.chat.completions.create(
        model="gpt-5",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Read this aloud:"},
                {"type": "input_audio", "input_audio": {"data": "...", "format": "wav"}}
            ]
        }],
        modalities=["text", "audio"],
        audio={"voice": "nova", "format": "opus"},
        stream=True
    )
    
    audio_chunks = []
    async for chunk in stream:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
        if chunk.choices[0].delta.audio:
            audio_chunks.append(chunk.choices[0].delta.audio.data)
    
    # Concatenate base64 chunks, decode once at the end
    full_audio_b64 = "".join(audio_chunks)
    return full_audio_b64

Pitfall: audio chunks are base64 fragments, not self-contained frames. You must concatenate them before decoding. Writing each chunk to a file independently produces corrupted audio.

Token accounting across modalities

Input tokens now include image tiles and audio frames. Output tokens include generated audio frames at roughly 1 token per 24ms of audio (Opus) or 16ms (WAV). A 30-second response at 24kHz Opus is ~1,250 output tokens on top of text tokens.

This means a voice conversation accumulates tokens fast. A 5-minute dialogue with audio responses can exceed 15k output tokens. Set max_completion_tokens defensively.

{
  "model": "gpt-5",
  "messages": [...],
  "modalities": ["text", "audio"],
  "audio": {"voice": "alloy", "format": "opus"},
  "max_completion_tokens": 4096
}

Routing and fallback strategy

When you run multimodal workloads at scale, provider availability becomes a real concern. Vision and audio models have different capacity profiles than text-only models. A text fallback to a smaller model does not help when the request contains audio.

Design your routing layer to match modalities to available providers:

def select_provider(request_modalities: list[str]) -> str:
    """Return provider identifier based on modality requirements."""
    if "audio" in request_modalities:
        # Only providers with audio support
        return "provider_with_audio"
    elif "image" in request_modalities:
        return "provider_with_vision"
    return "default_text_provider"

If you use a gateway that honors client routing directives, you can pass the modality requirement in a header and let the gateway handle the match:

curl -X POST https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "X-Required-Modalities: audio,image" \
  -d '{"model": "gpt-5", "messages": [...], "modalities": ["text", "audio"]}'

The gateway forwards provider cache-control hints, so repeated image analysis requests can hit cached embeddings when the provider supports it.

Common failure modes

Audio format mismatch

Sending MP3 data with format: "wav" produces hallucinated transcriptions. The model does not validate the container. Validate on your side before sending.

import magic

def validate_audio(data: bytes, declared_format: str) -> bool:
    mime = magic.from_buffer(data, mime=True)
    expected = {"wav": "audio/wav", "mp3": "audio/mpeg", "opus": "audio/opus"}
    return mime == expected.get(declared_format)

Image URL timeouts

The model fetches image URLs synchronously during request processing. A slow CDN adds 5-30 seconds of latency before the first token. Download and re-host images yourself, or pass data URIs for images under 20MB.

Modality confusion in multi-turn conversations

If turn 1 includes an image and turn 2 asks “what about the red object?”, the model remembers the image. But if you drop the image from the history to save tokens, the reference breaks. Keep at least the last image-containing message in context, or explicitly re-describe the visual content in text.

Audio output truncation

max_completion_tokens applies to all output modalities combined. If you set 1024 tokens and the text response uses 800, you get ~224 tokens of audio (~5 seconds). The audio cuts off mid-sentence. Budget separately: estimate text tokens, subtract from limit, allocate remainder to audio.

Cost optimization patterns

Reuse image embeddings

If you analyze the same image across multiple prompts (e.g., a document page with different questions), send the image once in a system message, then reference it in subsequent turns. The provider may cache the vision encoder output.

{
  "messages": [
    {"role": "system", "content": [{"type": "image_url", "image_url": {"url": "..."}}]},
    {"role": "user", "content": "Extract all tables from the document."},
    {"role": "assistant", "content": "..."},
    {"role": "user", "content": "Now summarize the financial section."}
  ]
}

Downsample audio input

Telephony audio at 8kHz contains no information above 4kHz. Sending 48kHz WAV wastes 6x tokens. Resample to 16kHz Opus before sending — the model handles it natively and you cut input tokens by 75%.

ffmpeg -i input.wav -ar 16000 -c:a libopus -b:a 16k output.opus

Batch image analysis

If you have 50 screenshots to classify, do not send 50 separate requests. Pack 5-10 images per request with a structured prompt. The fixed per-request overhead (auth, routing, queue) amortizes across images.

{
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Classify each image as: login, dashboard, settings, error. Return JSON array."},
      {"type": "image_url", "image_url": {"url": "img1.png", "detail": "low"}},
      {"type": "image_url", "image_url": {"url": "img2.png", "detail": "low"}},
      {"type": "image_url", "image_url": {"url": "img3.png", "detail": "low"}}
    ]
  }]
}

Latency budgeting

Typical first-token latency breakdown for a mixed request (text + 1 image + 5s audio):

Audio output adds another 200-500ms before the first audio chunk streams. If your product requires sub-2s voice response, you need:

• Pre-warmed connections
• Data URI images (no download)
• 16kHz Opus input
• max_completion_tokens tuned low
• A provider with reserved capacity

Testing checklist

Before shipping a multimodal feature, verify:

1. Format validation — Reject mismatched audio containers at your API boundary
2. Token limits — Load test with max_completion_tokens at your production ceiling
3. Streaming decode — Verify audio chunk concatenation works for 30s, 60s, 120s responses
4. Fallback paths — Simulate provider degradation; confirm text-only fallback does not activate for audio requests
5. Cost guardrails — Alert on per-request token spend exceeding your threshold
6. Cache behavior — Confirm repeated image requests hit provider cache (check x-cache headers)

When not to use unified multimodal

The single-model approach trades specialization for convenience. Avoid it when:

• Transcription accuracy is critical — Dedicated ASR models (Whisper large-v3, Nova-2) still beat GPT-5 on noisy audio, accents, and domain vocabulary
• Vision tasks need pixel precision — OCR, defect detection, and measurement require specialized vision heads
• Latency is non-negotiable — A cascaded pipeline (ASR → LLM → TTS) with small models can hit 800ms end-to-end; GPT-5 audio-to-audio is 2-4s
• Cost per minute matters at scale — Per-token audio pricing adds up faster than per-minute ASR/TTS APIs

For general-purpose assistants, meeting summarizers, and prototype voice interfaces, the unified model wins on developer velocity. For production voice agents handling 10k+ minutes/day, the cascaded architecture still wins on economics and control.

Start with the unified endpoint. Measure your actual token consumption, latency distribution, and error rates. Migrate pieces to specialized models only when the data justifies it. The API surface is stable; your architecture should be too.