Integrating audio inputs into multimodal LLM API calls

Adding audio input multimodal llm api support to your stack means more than recording a clip and POSTing it. You need correct encoding, a message schema the model understands, and a client that survives provider outages. This guide gives a copy-paste path from raw audio file to parsed LLM response.

Step 1: Prepare and encode your audio

Most multimodal endpoints expect PCM WAV or MP3 at a specific sample rate. GPT-4o audio expects 16-bit PCM, 16–24 kHz, mono. Convert with ffmpeg:

ffmpeg -i input.m4a -ar 16000 -ac 1 -c:a pcm_s16le clip.wav

Then base64 the bytes. In Python:

import base64

with open("clip.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode("ascii")

print(len(audio_b64))  # sanity check, > 0

Keep clips under 10 MB; many providers cap audio size. If longer, split or transcribe first.

Capture live audio

If you record in-process, use sounddevice to pull a numpy buffer, then write WAV:

import sounddevice as sd
import numpy as np
import wave

sr = 16000
rec = sd.rec(int(5 * sr), samplerate=sr, channels=1, dtype="int16")
sd.wait()
with wave.open("live.wav", "wb") as w:
    w.setnchannels(1)
    w.setsampwidth(2)
    w.setframerate(sr)
    w.writeframes(rec.tobytes())

Choose the right container

WAV is safest for OpenAI-compatible audio input. MP3 works on some models but adds decode overhead. Avoid webm unless the provider documents it.

Step 2: Structure the multimodal message payload

The audio input multimodal llm api call sends audio inside the content array as an input_audio object. The schema for OpenAI-compatible servers:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What is the speaker saying about the deadline?" },
    {
      "type": "input_audio",
      "input_audio": {
        "data": "BASE64_STRING",
        "format": "wav"
      }
    }
  ]
}

Note the format field must match the actual bytes. If you send MP3, set "format": "mp3".

System prompts still apply

Prepend a system message to set tone or output format. Multimodal models honor it like text-only calls.

{
  "role": "system",
  "content": "You are a meeting assistant. Extract action items and deadlines."
}

Mixing with images

Because this is a multimodal gateway, you can append an image_url part in the same content array. The model aligns audio and visual context:

{
  "type": "image_url",
  "image_url": { "url": "https://example.com/diagram.png" }
}

Step 3: Call the audio input multimodal llm api

Use the OpenAI Python client pointed at any OpenAI-compatible gateway. For example, n4n.ai provides one OpenAI-compatible endpoint that addresses 240+ models and automatically falls back when a provider is rate-limited, so the same code works without vendor lock-in.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # or your provider's URL
    api_key="YOUR_KEY",
)

resp = client.chat.completions.create(
    model="gpt-4o-audio-preview",  # or any audio-capable model
    messages=[
        {"role": "system", "content": "You are a meeting assistant."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Summarize this audio."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio_b64, "format": "wav"}
                }
            ]
        }
    ],
    max_tokens=300,
)

print(resp.choices[0].message.content)

If you prefer raw requests:

import requests

payload = {
    "model": "gpt-4o-audio-preview",
    "messages": [
        {"role": "user", "content": [
            {"type": "text", "text": "Transcribe and translate to English."},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}}
        ]}
    ]
}
r = requests.post(
    "https://api.n4n.ai/v1/chat/completions",
    headers={"Authorization": "Bearer YOUR_KEY"},
    json=payload,
)
print(r.json()["choices"][0]["message"]["content"])

Pick a model that actually accepts audio

Not every LLM does. Confirm the model card: GPT-4o variants, Gemini 1.5 Pro/Flash, and some open weights support audio. If you request audio on a text-only model, you’ll get a 400 with a clear error.

Step 4: Parse the response and meter usage

The completion returns text (or audio if you requested output_audio). Extract content and token counts:

content = resp.choices[0].message.content
usage = resp.usage
print(f"Prompt tokens: {usage.prompt_tokens}, Completion: {usage.completion_tokens}")

Per-token usage metering lets you attribute cost. If you send through a gateway, the usage object reflects the upstream provider’s accounting.

Streaming audio input

You can stream the response by adding stream=True. The request body is identical; iterate chunks:

stream = client.chat.completions.create(
    model="gpt-4o-audio-preview",
    messages=messages,
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Streaming doesn’t change audio encoding on the input side.

Step 5: Add resilience with routing directives

Production systems hit rate limits. Some gateways let you pin a provider or set cache hints via headers or extra body fields. For example, pass route: { "provider": "openai" } if your client supports it, or set cache_control on the audio part to leverage provider-side caching of identical clips. Gateways such as n4n.ai honor client routing directives and forward provider cache-control hints, letting you mark repeated audio with cache_control: ephemeral to cut cost.

Implement a simple retry with backoff client-side as well:

import time

def call_with_retry(messages, attempts=3):
    for i in range(attempts):
        try:
            return client.chat.completions.create(
                model="gpt-4o-audio-preview",
                messages=messages,
            )
        except Exception as e:
            if i == attempts - 1:
                raise
            time.sleep(2 ** i)

Step 6: Verify and test the integration

To confirm the audio input multimodal llm api integration works end to end:

1. Use a 3-second clip with clear speech, e.g., “The deploy is scheduled for Friday at noon.”
2. Send with prompt: “Repeat the time and day mentioned.”
3. Assert the response contains “Friday” and “noon”.
4. Check usage.prompt_tokens is non-zero and scales with audio length.

If the model echoes the content correctly, your pipeline is solid. For negative testing, send a silent clip and confirm the model says no speech detected or asks for clarification.

Common failures

• 400: invalid audio format — base64 padding or wrong format string.
• 413: payload too large — compress or trim.
• 422: model does not support audio — switch model.

Wrapping up

You now have a working path to send audio into a multimodal LLM: encode to WAV, embed as input_audio, call an OpenAI-compatible endpoint, and parse the text. Swap models or providers without rewriting the payload. The audio input multimodal llm api pattern stays consistent across the major vendors, so invest in a clean encoding helper and reuse it.