Analyze images and PDFs with Gemini 2.0 in LangChain

Gemini 2.0 LangChain image PDF analysis is straightforward once you stop treating multimodal input as a special case and just pass base64 blobs to the model. This tutorial builds a working pipeline that extracts structured insight from a scanned diagram and a multi-page PDF using ChatGoogleGenerativeAI, with no external OCR or document-splitting services required.

Prerequisites

• Python 3.10 or newer
• A Google AI Studio key with Gemini 2.0 access (GOOGLE_API_KEY in env)
• langchain-google-genai and python-dotenv installed
• A local image (PNG/JPEG) and a PDF you want to inspect

pip install langchain-google-genai python-dotenv

Keep the sample files small for this walkthrough: a screenshot under 5 MB and a PDF under 10 MB. Gemini 2.0 parses PDFs natively, so you do not need pdfplumber or image conversion steps.

Initialize the chat model

Use the Flash variant for low-latency analysis; switch to Pro if you need deeper reasoning on complex contracts.

import os
from dotenv import load_dotenv
from langchain_google_genai import ChatGoogleGenerativeAI

load_dotenv()

llm = ChatGoogleGenerativeAI(
    model="gemini-2.0-flash",
    google_api_key=os.getenv("GOOGLE_API_KEY"),
    temperature=0.1,
)

The temperature near zero keeps extractions deterministic, which matters when you pipe the output into downstream validators.

Load files as base64

LangChain’s Google integration accepts inline data through standard content blocks. Encode the raw bytes once.

import base64

def b64_path(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

img_b64 = b64_path("diagram.png")
pdf_b64 = b64_path("report.pdf")

Do not base64-encode twice or prepend data: URIs for the file block; the mime type is passed separately.

Analyze an image

Construct a HumanMessage with a text instruction and an image_url block. The data: URI scheme is required for inline images.

from langchain_core.messages import HumanMessage

img_msg = HumanMessage(content=[
    {
        "type": "text",
        "text": "Extract the chart title, axis labels, and the three highest data points as JSON.",
    },
    {
        "type": "image_url",
        "image_url": {"url": f"data:image/png;base64,{img_b64}"},
    },
])

img_resp = llm.invoke([img_msg])
print(img_resp.content)

Expected output resembles:

{
  "title": "Q4 Inference Latency",
  "x_axis": "Region",
  "y_axis": "p95 ms",
  "top_points": [
    {"region": "eu-west", "value": 142},
    {"region": "us-east", "value": 118},
    {"region": "ap-south", "value": 201}
  ]
}

If you get a raw text dump instead of JSON, tighten the prompt; Gemini 2.0 follows structured instructions well but won’t force a schema unless you ask.

Analyze a PDF

The file content block is the correct way to hand a PDF to the model. No page rendering required.

pdf_msg = HumanMessage(content=[
    {
        "type": "text",
        "text": "Summarize this document in 3 bullets and list any dates mentioned.",
    },
    {
        "type": "file",
        "file": {"mime_type": "application/pdf", "data": pdf_b64},
    },
])

pdf_resp = llm.invoke([pdf_msg])
print(pdf_resp.content)

Expected output:

- Vendor agreement for GPU cluster rental, effective 2025-01-01.
- Liability capped at monthly fees; indemnity clause excludes gross negligence.
- Termination allowed with 30 days notice after initial 12-month term.
Dates: 2025-01-01, 2025-12-31, 2026-01-30.

Gemini 2.0 LangChain image PDF analysis benefits from the model’s 1M-token context: you can pass a 50-page PDF and ask cross-page questions without chunking.

Multi-document reasoning

You can combine both blocks in one turn. This is where the native multimodal path beats gluing together separate vision and text calls.

combined = HumanMessage(content=[
    {"type": "text", "text": "The image is the executive summary chart. The PDF is the full report. Do the chart numbers match the PDF's table 2?"},
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}},
    {"type": "file", "file": {"mime_type": "application/pdf", "data": pdf_b64}},
])

combined_resp = llm.invoke([combined])
print(combined_resp.content)

The model returns a reconciliation note citing specific values. In practice, we’ve seen it catch a 12% discrepancy that a regex table extractor missed because the PDF used merged cells.

Streaming for UX

Long PDF summaries should stream. The API is identical except for iteration.

for chunk in llm.stream([pdf_msg]):
    print(chunk.content, end="", flush=True)

Streaming does not change token billing; you still pay per output token.

Routing and fallback in production

If you deploy this beyond a notebook, key management and provider outages become real. An OpenAI-compatible gateway such as n4n.ai exposes Gemini 2.0 alongside 240+ models behind one endpoint, with automatic fallback when a provider is rate-limited or degraded, and forwards provider cache-control hints so repeated PDF analyses hit context cache. You can swap ChatGoogleGenerativeAI for ChatOpenAI pointing at that endpoint without rewriting the message blocks above, since the file and image_url types map cleanly.

Practical limits and error handling

• Inline payloads are bounded by Gemini’s request size limit (tens of MB). For larger PDFs, use the File API to upload and pass a file_uri instead of base64.
• Always wrap invoke in try/except; GoogleGenerativeAIError surfaces quota and mime-type mismatches with actionable messages.
• Set max_output_tokens explicitly when you need bounded responses for parsing.
• If the PDF is scanned image-only, Gemini 2.0 still runs OCR internally—but accuracy on native text layers is higher.

Gemini 2.0 LangChain image PDF analysis replaces a stack of specialized libraries with one model call. Ship the happy path first, then add the File API and streaming once the prompt is stable.