How Gemini 3 processes video input natively

Gemini 3 processes video natively by breaking frames into visual tokens that share the same vocabulary space as text. This means you send video directly to the model — no external frame extraction, no separate vision encoder pipeline, no stitching together caption outputs. The model sees video as a sequence of tokens interleaved with your prompt, just like a long document. Understanding how that tokenization works, where the limits bite, and how to structure requests for cost and latency control is what separates a demo from a production feature.

How video tokenization actually works

When you send a video to Gemini 3, the model samples frames at a fixed rate (1 frame per second by default) and encodes each frame into a grid of visual tokens. Each token represents a 16×16 pixel patch after the frame is resized to a maximum dimension — typically 1024×1024 for the 1.5 Pro model. A 720p frame at 1280×720 becomes roughly 3,600 tokens. At 1 fps, a 60-second video consumes approximately 216,000 tokens before you add a single word of prompt.

The token budget is shared across text and video. Gemini 3.5 Pro offers a 2 million token context window, but video eats it fast. A 10-minute 1080p clip at 1 fps is roughly 1.3 million tokens. You have headroom for the prompt and response, but not for much else. The 1.5 Flash model has a 1 million token window and lower per-token cost, making it the default choice for high-volume video workloads where you don’t need Pro’s reasoning depth.

You can control frame sampling through the video_metadata parameter. The API accepts start_offset, end_offset, and fps fields. Setting fps: 0.5 halves token consumption with minimal quality loss for slow-moving content like lectures or security footage. For action-heavy video — sports, gaming, UI walkthroughs — you’ll want 1 fps or higher. There’s no adaptive sampling; the model doesn’t detect scene changes. You decide the tradeoff.

{
  "contents": [{
    "role": "user",
    "parts": [
      {"text": "Summarize the key decisions made in this meeting."},
      {
        "file_data": {
          "mime_type": "video/mp4",
          "file_uri": "https://storage.googleapis.com/bucket/meeting.mp4"
        },
        "video_metadata": {
          "start_offset": "0s",
          "end_offset": "1800s",
          "fps": 0.5
        }
      }
    ]
  }]
}

Uploading video: Files API vs inline data

For anything over 20 MB, use the Files API. Inline base64 encoding bloats request payloads, hits HTTP size limits, and forces re-upload on every retry. The Files API gives you a resumable upload, a stable file_uri, and automatic cleanup after 48 hours (configurable up to 30 days).

from google import genai

client = genai.Client(api_key="YOUR_API_KEY")

# Upload once, reference many times
video_file = client.files.upload(
    file="meeting_recording.mp4",
    config={"mime_type": "video/mp4"}
)

# Wait for processing — required before the model can read it
while video_file.state.name == "PROCESSING":
    time.sleep(2)
    video_file = client.files.get(name=video_file.name)

response = client.models.generate_content(
    model="gemini-1.5-pro",
    contents=[
        "Extract all action items with timestamps.",
        video_file
    ]
)

The Files API also lets you reuse the same video across multiple prompts — useful for multi-turn analysis or running different extraction passes (action items, then decisions, then sentiment) without re-uploading. Each prompt still pays the full token cost for the video, but you save upload bandwidth and latency.

Pitfall: The file must be in ACTIVE state before you call generate_content. The upload response returns immediately; processing happens asynchronously. Poll client.files.get() until state.name == "ACTIVE". Skipping this check is the most common cause of “file not found” errors that look like permission problems.

Structuring prompts for video

Video prompts work best when you give the model explicit structure. Unlike text, where the model can scan bidirectionally, video tokens are consumed sequentially. The model “watches” from start to end. If your question requires information from the last 30 seconds of a 20-minute video, the model must process everything before it.

Put the task instruction before the video in the parts array. This primes the model’s attention as it processes frames. For extraction tasks, ask for structured output — JSON with timestamps — rather than free text. The model is surprisingly good at emitting valid JSON when you provide a schema.

from pydantic import BaseModel
from typing import List

class ActionItem(BaseModel):
    timestamp: str  # MM:SS format
    owner: str
    priority: str  # high, medium, low

response = client.models.generate_content(
    model="gemini-1.5-pro",
    contents=[
        "Extract action items as JSON matching this schema: "
        + ActionItem.model_json_schema(),
        video_file
    ],
    config={
        "response_mime_type": "application/json",
        "response_schema": ActionItem.model_json_schema(),
    }
)

items: List[ActionItem] = response.parsed

The response_schema parameter (available in the Python SDK) enforces structure at generation time. You get typed objects back, not strings you have to parse. This also reduces hallucinated fields — the model literally cannot emit keys that aren’t in the schema.

Tradeoff: Structured output adds latency. For high-throughput pipelines where you process thousands of videos daily, consider a two-pass approach: first pass with a lightweight prompt on Flash to get rough timestamps, second pass on Pro with structured output only on the relevant segments.

Handling long video: segmentation and stitching

When video exceeds the context window — or when you want to parallelize — split it into overlapping chunks. Overlap by 10–15 seconds so context doesn’t get lost at boundaries. Process chunks in parallel, then stitch results.

def segment_video(video_path: str, chunk_duration: int = 600, overlap: int = 15) -> List[dict]:
    """Returns list of {start, end, file_uri} for each chunk."""
    # Use ffmpeg to split - this is pseudo-code, adapt to your infra
    chunks = []
    duration = get_video_duration(video_path)
    start = 0
    while start < duration:
        end = min(start + chunk_duration, duration)
        chunk_uri = upload_chunk(video_path, start, end)
        chunks.append({"start": start, "end": end, "uri": chunk_uri})
        start += chunk_duration - overlap
    return chunks

def process_chunks_parallel(chunks: List[dict], prompt: str) -> List[dict]:
    with ThreadPoolExecutor(max_workers=8) as executor:
        futures = [
            executor.submit(analyze_chunk, chunk["uri"], prompt, chunk["start"])
            for chunk in chunks
        ]
        return [f.result() for f in as_completed(futures)]

def analyze_chunk(uri: str, prompt: str, offset_seconds: int) -> dict:
    response = client.models.generate_content(
        model="gemini-1.5-flash",
        contents=[prompt, {"file_data": {"mime_type": "video/mp4", "file_uri": uri}}]
    )
    # Adjust timestamps by offset_seconds
    return {"offset": offset_seconds, "data": response.text}

Stitching is domain-specific. For transcripts, concatenate with timestamp adjustment. For summaries, feed chunk summaries into a final synthesis prompt. For entity extraction, deduplicate by (entity, normalized_timestamp).

Pitfall: Overlap doesn’t fix hard cuts. If a speaker starts a sentence in chunk N and finishes in chunk N+1, neither chunk has the full utterance. For transcription-critical workloads, run a separate ASR pass (Whisper, Chirp) on the audio track and use Gemini only for understanding — not transcription.

Audio track handling

Gemini 3 processes the audio track natively alongside frames. You don’t need to extract audio separately. The model hears speech, background noise, music, and non-speech audio events. This is a significant advantage over frame-only approaches: a presenter saying “as you can see on this slide” while pointing at a chart is intelligible only with audio.

However, audio consumes tokens too — roughly 1,000 tokens per 15 seconds of audio at the model’s internal sampling rate. For a 30-minute video, that’s ~120,000 tokens on top of the visual tokens. If your task is purely visual (UI walkthrough, silent security cam), you can’t currently disable audio processing. The workaround: strip audio before upload with ffmpeg -an.

ffmpeg -i input.mp4 -an -c:v copy silent.mp4

This reduces token count by 5–10% depending on video length. Worth it at scale.

Cost and latency modeling

Token pricing (as of writing): Gemini 1.5 Pro is $3.50/M input tokens, Flash is $0.075/M. A 10-minute 720p video at 1 fps is ~216K visual tokens + ~40K audio tokens = ~256K tokens. On Pro: ~$0.90 per video. On Flash: ~$0.02. Latency: Pro averages 15–30 seconds for this size; Flash 5–12 seconds.

Batch processing (the batchGenerateContent endpoint) cuts Flash latency by ~40% and Pro by ~25% at 50% cost. Use it for any workload where you don’t need sub-minute turnaround.

# Batch API - pseudo-code, check current SDK
batch = client.batches.create(
    model="gemini-1.5-flash",
    requests=[
        {"contents": [prompt, {"file_data": {"file_uri": uri}}]}
        for uri in video_uris
    ]
)
# Poll batch.get() until complete, then download results

Reality check: The 2M token window is theoretical. In practice, videos over ~1.5M tokens (about 12 minutes at 1080p/1fps) start hitting quality degradation on Pro — the model loses early context. Flash degrades earlier, around 800K tokens. Plan your chunking strategy accordingly.

Common failure modes

1. “Video too long” error on upload: The Files API rejects files over 2 GB. Compress or segment before upload. H.264 at 2–4 Mbps for 720p is usually sufficient for model understanding.

2. Timestamp hallucination: The model estimates timestamps from frame position, not from a timecode track. At 1 fps, it can be off by ±1 second. At 0.5 fps, ±2 seconds. For precise timestamps, cross-reference with an ASR transcript.

3. Language switching mid-video: If a video switches languages, the model handles it but may miss the transition point. Prompt explicitly: “This video contains English and Spanish segments. Note language changes with timestamps.”

4. Text in video too small: Slides, code, UI text below ~24pt at 720p often become unreadable after the model’s internal resize. If your use case depends on reading on-screen text, pre-extract frames at native resolution and send as images (higher token cost, but legible).

5. Rate limits on Files API: Upload quota is separate from generation quota. If you’re processing hundreds of videos/hour, you’ll hit upload limits before generation limits. Implement exponential backoff and consider a staging bucket with a worker pool.

Production patterns

Pattern: Pre-filter with cheap model Run a 5-second clip through Flash with “Does this video contain X?” before committing full processing. Saves 90%+ cost on negative cases.

Pattern: Cache frequent videos If users re-analyze the same uploaded videos (common in collaborative tools), cache the file_uri and reuse. The Files API respects If-None-Match headers for conditional requests.

Pattern: Async pipeline with webhooks Upload → process → store results → webhook client. Never block a user request on video generation. Even Flash at 5 seconds is too slow for synchronous HTTP.

Pattern: Fallback to n4n.ai for multi-provider resilience When Gemini hits quota or regional outages, routing the same request to an alternative provider (via an OpenAI-compatible endpoint) keeps the pipeline moving. The request format stays identical; only the base URL and auth header change.

What’s not supported (yet)

• Variable frame rate sampling (adaptive to motion)
• Region-of-interest cropping before tokenization
• Streaming video input (WebRTC, RTMP)
• Multiple videos in a single request (you can send multiple file_data parts, but they concatenate sequentially — no cross-video attention)
• Video output generation

The roadmap suggests some of these arrive in 2025. For now, design around the constraints.

Quick reference: token estimation cheat sheet

Multiply by 1.05–1.10 for audio tokens. Add 2–5K for prompt/response overhead.

Start with Flash at 0.5 fps for exploration. Move to Pro only when reasoning quality demands it. Segment aggressively. Structure your output. Monitor token usage per video — it’s the single strongest predictor of both cost and latency.