How multimodal models combine text, image, and audio

Multimodal models text image audio processing has moved from research demos to production infrastructure. If you’re building features that need to understand a screenshot alongside a user’s question, or transcribe a voice note while extracting entities from the accompanying text, you need to know how these models actually fuse representations — and where the sharp edges are. This guide walks through the architecture, tokenization, and integration patterns you’ll encounter when shipping multimodal features.

How the major architectures fuse modalities

Most production multimodal models follow one of three fusion strategies. The choice determines latency, context window pressure, and how you handle streaming.

Early fusion (joint embedding space)

Models like GPT-4o, Gemini 1.5, and Qwen2-VL project all modalities into a shared token vocabulary before the transformer sees them. Images become sequences of visual tokens (typically 256–1024 per image depending on resolution), audio becomes acoustic tokens, and text stays as text tokens. The transformer attends over the concatenated sequence uniformly.

# Conceptual early-fusion input construction
def build_multimodal_sequence(text: str, image: Image, audio: Audio) -> list[int]:
    text_tokens = tokenizer.encode(text)
    image_tokens = vision_encoder.encode(image)  # e.g., 576 tokens for 336x336
    audio_tokens = audio_encoder.encode(audio)   # e.g., 150 tokens for 5s @ 16kHz
    return [BOS] + image_tokens + [IMG_SEP] + audio_tokens + [AUD_SEP] + text_tokens + [EOS]

Tradeoff: simple attention mechanism, but visual/audio tokens consume massive context. A single high-res image can burn 2,000+ tokens. You pay for every pixel whether it’s relevant or not.

Late fusion (separate encoders, cross-attention)

Models like Flamingo, IDEFICS, and some open-weight variants keep modality-specific encoders frozen and inject compressed representations via cross-attention layers inserted into a pretrained LLM backbone. The image encoder outputs a fixed small set of tokens (often 64–256) regardless of input resolution.

# Late fusion: image encoder -> projector -> cross-attention in LLM
class LateFusionModel(nn.Module):
    def __init__(self, llm, vision_encoder, projector):
        self.llm = llm
        self.vision_encoder = vision_encoder
        self.projector = projector  # maps vision dim -> llm dim
        
    def forward(self, input_ids, images):
        # Vision path: fixed compute regardless of image size
        vision_features = self.vision_encoder(images)  # [B, 256, 1024]
        vision_tokens = self.projector(vision_features)  # [B, 64, 4096]
        
        # LLM with cross-attention layers sees vision_tokens as KV
        return self.llm(input_ids, cross_attn_kv=vision_tokens)

Tradeoff: constant vision token budget, easier to fit in context. But the fixed bottleneck can lose fine-grained visual detail (small text in images, precise spatial relationships). Cross-attention layers also add parameters you may need to fine-tune.

Hybrid fusion (hierarchical or mixture-of-experts)

Newer models like LLaVA-Next, InternVL2, and Molmo use a vision encoder that outputs multi-scale features, then a connector (MLP, Q-Former, or downsampling attention) that adapts token count to resolution. Some route different modalities through specialized experts.

# Hybrid: adaptive visual tokens based on resolution
def adaptive_vision_tokens(image: Image, max_tokens: int = 1024) -> torch.Tensor:
    patches = vit_patchify(image)  # [N_patches, 768]
    # Downsample or pool to fit budget
    if len(patches) > max_tokens:
        patches = adaptive_pool(patches, max_tokens)
    return connector(patches)  # [actual_tokens, llm_dim]

Tradeoff: best of both worlds on paper, but connector training is finicky. Resolution changes can shift token positions, breaking positional assumptions in the LLM.

Tokenization strategies you’ll actually debug

Understanding how your provider tokenizes multimodal input determines your prompt engineering, cost model, and context management.

Image tokenization: patches, thumbnails, and dynamic resolution

Most vision encoders (ViT, SigLIP, CLIP) split images into fixed-size patches (14×14 or 16×16). A 336×336 image at 14×14 patches yields 576 tokens. But production systems use dynamic resolution:

# Typical dynamic resolution bucketing (like LLaVA-Next)
RESOLUTION_BUCKETS = [
    (336, 336),    # 576 tokens
    (672, 336),    # 1152 tokens  
    (336, 672),    # 1152 tokens
    (672, 672),    # 2304 tokens
    (1008, 336),   # 1728 tokens
]

def select_bucket(width: int, height: int, max_tokens: int) -> tuple[int, int]:
    # Pick largest bucket under token budget preserving aspect ratio
    for w, h in sorted(RESOLUTION_BUCKETS, key=lambda x: x[0]*x[1], reverse=True):
        tokens = (w // 14) * (h // 14)
        if tokens <= max_tokens:
            return (w, h)
    return RESOLUTION_BUCKETS[0]

Pitfall: providers often don’t document their bucketing. You send a 1920×1080 screenshot and get charged for 4,000+ tokens because it got upscaled to the nearest bucket. Always test with actual API calls and log usage.prompt_tokens.

Audio tokenization: codec tokens vs. semantic tokens

Audio falls into two camps. Codec models (SoundStream, Encodec, DAC) compress waveforms into discrete tokens at 50–75 Hz — a 10-second clip becomes 500–750 tokens. Semantic models (Whisper encoder, HuBERT, WavLM) output frame-level embeddings that a projector downsamples.

# Codec tokenization (what you send to GPT-4o audio)
def audio_to_codec_tokens(waveform: np.ndarray, sample_rate: int) -> list[int]:
    # Resample to 24kHz, encode with DAC/SoundStream
    waveform = resample(waveform, sample_rate, 24000)
    codes = dac_model.encode(waveform)  # [n_codebooks, n_frames]
    # Flatten codebooks into single token stream (interleaved or concatenated)
    return flatten_codes(codes)  # ~75 tokens/second

# Semantic tokenization (what you might use with open models)
def audio_to_semantic_tokens(waveform: np.ndarray) -> torch.Tensor:
    features = whisper_encoder(waveform)  # [1, 1500, 1024] for 30s
    # Projector downsamples 6x -> 250 tokens
    return audio_projector(features)

Tradeoff: codec tokens preserve reconstruction fidelity (you can decode back to audio) but need more tokens. Semantic tokens are compact but lossy — fine for understanding, useless for generation.

Text stays text, but watch the special tokens

Multimodal models insert modality boundary tokens (<|image|>, <|audio|>, <|video|>) that count against your context. Some APIs charge for these; others don’t. The tokenizers also differ in how they handle interleaving:

// OpenAI-style interleaved content (what you send)
{
  "role": "user",
  "content": [
    {"type": "text", "text": "What's in this image?"},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
    {"type": "text", "text": "And this one?"},
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
  ]
}

// What the model actually sees (tokenized)
[BOS] "What's in this image?" [IMG_START] <576 vision tokens> [IMG_END] 
"And this one?" [IMG_START] <576 vision tokens> [IMG_END] [EOS]

Pitfall: some open models expect a specific template like <image>\n{text} and break silently if you omit the newline. Always verify the chat template against the model’s tokenizer config.

Building the request pipeline

Production multimodal pipelines need to handle encoding, validation, fallbacks, and streaming. Here’s a practical structure.

Input normalization and validation

# multimodal/types.py
from dataclasses import dataclass
from enum import Enum
from typing import Optional, Union
import base64

class Modality(Enum):
    TEXT = "text"
    IMAGE = "image"
    AUDIO = "audio"

@dataclass
class MediaInput:
    modality: Modality
    data: Union[str, bytes]  # base64 string, raw bytes, or URL
    mime_type: Optional[str] = None
    # Metadata for routing / cost estimation
    width: Optional[int] = None
    height: Optional[int] = None
    duration_seconds: Optional[float] = None

def normalize_input(raw: dict) -> MediaInput:
    """Accept flexible input formats, return validated MediaInput."""
    modality = Modality(raw["type"])
    
    if modality == Modality.IMAGE:
        # Handle data URI, base64, raw bytes, or HTTP URL
        data = raw["image_url"]["url"] if "image_url" in raw else raw["data"]
        mime, b64 = parse_data_uri(data) if data.startswith("data:") else (None, data)
        img = decode_image(b64)  # PIL Image
        return MediaInput(
            modality=Modality.IMAGE,
            data=b64,
            mime_type=mime or "image/png",
            width=img.width,
            height=img.height
        )
    
    elif modality == Modality.AUDIO:
        data = raw["input_audio"]["data"] if "input_audio" in raw else raw["data"]
        mime = raw["input_audio"].get("format", "wav")
        duration = probe_audio_duration(data, mime)
        return MediaInput(
            modality=Modality.AUDIO,
            data=data,
            mime_type=f"audio/{mime}",
            duration_seconds=duration
        )
    
    return MediaInput(modality=Modality.TEXT, data=raw["text"])

Cost-aware routing and fallback

Different providers price multimodal tokens differently. Some charge per image (flat), others per token. Audio is often priced per second of input. Build a router that estimates cost before sending:

# multimodal/router.py
from dataclasses import dataclass
from typing import Literal

@dataclass
class ModelCost:
    model: str
    text_per_1k: float
    image_per_token: float  # or per_image if flat
    audio_per_second: float
    max_context: int
    supports_streaming: bool

PROVIDER_COSTS = {
    "gpt-4o": ModelCost("gpt-4o", 5.00, 0.00125, 0.06, 128000, True),
    "gpt-4o-mini": ModelCost("gpt-4o-mini", 0.15, 0.0001875, 0.024, 128000, True),
    "claude-3-5-sonnet": ModelCost("claude-3-5-sonnet", 3.00, 0.0015, 0.0, 200000, True),
    "gemini-1.5-pro": ModelCost("gemini-1.5-pro", 3.50, 0.00075, 0.0, 2000000, True),
}

def estimate_cost(inputs: list[MediaInput], model: str) -> float:
    cost = PROVIDER_COSTS[model]
    total = 0.0
    for inp in inputs:
        if inp.modality == Modality.TEXT:
            tokens = estimate_tokens(inp.data)
            total += (tokens / 1000) * cost.text_per_1k
        elif inp.modality == Modality.IMAGE:
            tokens = estimate_image_tokens(inp.width, inp.height)
            total += tokens * cost.image_per_token
        elif inp.modality == Modality.AUDIO:
            total += (inp.duration_seconds or 0) * cost.audio_per_second
    return total

def select_model(inputs: list[MediaInput], budget_usd: float, require_streaming: bool) -> str:
    candidates = [m for m, c in PROVIDER_COSTS.items() 
                  if c.supports_streaming == require_streaming]
    # Pick cheapest that fits context and budget
    for model in sorted(candidates, key=lambda m: estimate_cost(inputs, m)):
        if estimate_cost(inputs, model) <= budget_usd:
            return model
    raise ValueError("No model fits budget")

Streaming with multimodal context

Streaming works differently when the prompt includes images or audio. The model must process all visual/audio tokens before generating the first text token. This creates a “prefill latency” spike:

# multimodal/streaming.py
import asyncio
import time
from openai import AsyncOpenAI

async def stream_multimodal(
    client: AsyncOpenAI,
    messages: list[dict],
    model: str,
    on_first_token: callable = None
) -> AsyncGenerator[str, None]:
    """Stream with prefill latency tracking."""
    start = time.perf_counter()
    first_token_time = None
    
    stream = await client.chat.completions.create(
        model=model,
        messages=messages,
        stream=True,
        stream_options={"include_usage": True}
    )
    
    async for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.perf_counter()
                prefill_latency = first_token_time - start
                if on_first_token:
                    on_first_token(prefill_latency)
            yield chunk.choices[0].delta.content
    
    # Log usage for cost tracking
    if chunk.usage:
        log_usage(model, chunk.usage.prompt_tokens, chunk.usage.completion_tokens)

Pitfall: with large images, prefill can take 2–10 seconds. Users see nothing during this time. Show a “processing image…” indicator, not a blank streaming cursor. Consider splitting: send image first with a “describe this” prompt, cache the description, then stream the actual conversation.

Common failure modes and how to detect them

Modality hallucination

Models confidently describe objects that aren’t in the image, or transcribe words that weren’t in the audio. This happens most when:

• Image resolution is too low for the detail asked about (reading small text, counting objects)
• Audio has background noise, overlapping speakers, or accent mismatch
• The prompt asks for reasoning beyond the modality’s capacity (“what is the person thinking?”)

Detection: run a consistency check. For images, ask the same question with different phrasings and compare. For audio, run ASR separately (Whisper) and compare transcriptions.

# multimodal/validation.py
async def validate_image_response(
    client: AsyncOpenAI,
    image_b64: str,
    question: str,
    n_samples: int = 3
) -> dict:
    """Run multiple samples, return consensus + variance."""
    responses = []
    for _ in range(n_samples):
        resp = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": question},
                    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}}
                ]
            }],
            temperature=0.3,
            max_tokens=200
        )
        responses.append(resp.choices[0].message.content)
    
    # Simple heuristic: if answers disagree on key entities, flag
    return {
        "responses": responses,
        "consensus": max(set(responses), key=responses.count) if responses else None,
        "agreement_ratio": responses.count(max(set(responses), key=responses.count)) / n_samples
    }

Context window exhaustion from visual tokens

A common production bug: conversation history includes previous images, each consuming 1,000+ tokens. After 5 turns you’ve blown 80% of context on stale images.

Fix: implement a multimodal-aware context manager that drops old images first, summarizes them to text, or uses a sliding window with image-aware scoring.

# multimodal/context.py
def trim_multimodal_history(
    messages: list[dict],
    max_tokens: int,
    tokenizer,
    image_token_cost: int = 1000
) -> list[dict]:
    """Drop oldest images first, then oldest text."""
    # Count tokens per message
    msg_tokens = []
    for msg in messages:
        tokens = 0
        for content in msg.get("content", []) if isinstance(msg.get("content"), list) else [msg.get("content", "")]:
            if isinstance(content, dict) and content.get("type") == "image_url":
                tokens += image_token_cost
            elif isinstance(content, str):
                tokens += len(tokenizer.encode(content))
        msg_tokens.append((msg, tokens))
    
    # Trim from oldest, preferring to drop image-heavy messages
    total = sum(t for _, t in msg_tokens)
    while total > max_tokens and msg_tokens:
        # Find oldest message with images
        for i, (msg, tokens) in enumerate(msg_tokens):
            has_image = any(
                isinstance(c, dict) and c.get("type") == "image_url"
                for c in (msg.get("content", []) if isinstance(msg.get("content"), list) else [])
            )
            if has_image:
                total -= tokens
                msg_tokens.pop(i)
                break
        else:
            # No images left, drop oldest text
            _, tokens = msg_tokens.pop(0)
            total -= tokens
    
    return [msg for msg, _ in msg_tokens]

Provider-specific quirks

• OpenAI: gpt-4o accepts image_url with detail: "low" (85 tokens) or detail: "high" (variable, up to ~4k). gpt-4o-mini only supports low. Audio input requires input_audio format with base64 PCM16 at 24kHz.
• Anthropic: Images must be base64 (no URLs). Max 20 images per request. No audio input support yet.
• Google: Gemini accepts inline base64 or File API URIs. Video counts as 1 token/second. Audio input via inline_data with MIME type.
• Open weights (vLLM, TGI): Chat templates vary wildly. LLaVA-Next uses <image>\n, Qwen2-VL uses <|vision_start|>...<|vision_end|>, Molmo uses <image> with no newline. Always load the tokenizer’s chat_template and render with it.

Evaluation checklist before shipping

Run these checks on your specific data, not benchmarks:

1. Resolution sweep: Test your task at each resolution bucket your provider supports. Plot accuracy vs. token cost. You’ll often find a sweet spot at 672×672 or 1008×336.
2. Audio quality ladder: Test with clean speech, phone-quality (8kHz), noisy background, and accented speakers. Measure WER degradation.
3. Interleaving stress: Feed 5+ images in one request. Verify the model associates each answer with the correct image.
4. Streaming prefill: Measure time-to-first-token at 1, 3, 5, 10 images. Build your UX around the P99.
5. Cost per successful task: Not cost per token. Include retries, validation calls, and fallback attempts.
6. Fallback behavior: Simulate provider degradation. Does your router switch to a cheaper model gracefully? Does quality drop acceptably?

Where to go deeper

• Architecture papers: “LLaVA: Large Language and Vision Assistant” (early fusion), “Flamingo: a Visual Language Model for Few-Shot Learning” (late fusion), “InternVL: Scaling up Vision Foundation Models” (hybrid).
• Tokenization: DAC paper for audio codec tokens, SigLIP/CLIP papers for vision patch embeddings.
• Production patterns: vLLM’s multimodal support (request-level parallelism for prefill), TGI’s prefill chunking for long visual contexts.

The field moves fast — new connectors, new tokenizers, new fusion strategies every quarter. But the engineering constraints stay the same: context budgets, latency budgets, cost budgets, and the gap between benchmark prompts and your users’ actual inputs. Build observability around those four dimensions and you’ll survive the next architecture shift.