Sending PDFs and documents to multimodal LLM APIs

Most teams hit a wall when they try to send pdf documents multimodal api calls: the vision endpoints expect images or base64 text, not raw PDF bytes. This guide walks through extracting pages, rendering them to PNG, and posting a correctly shaped request to an OpenAI-compatible chat completion endpoint.

Step 1: Pick an extraction strategy

You cannot upload a .pdf to a multimodal vision model and expect it to “just read” the file. The API surface accepts image URLs or base64 image data, plus text. Your job is to translate the PDF into one of those inputs.

Two paths exist:

• Render pages to images for scanned docs or layouts where spatial structure matters (tables, forms, signatures).
• Extract text with a parser (pdfminer, PyMuPDF) and send as a text block when the document is native digital text and you only need content.

If you need to send pdf documents multimodal api requests for a mixed corpus, render to images. Vision models handle rendered text well and you avoid parser edge cases like broken paragraph detection. For purely textual PDFs, text extraction is cheaper because it consumes fewer tokens than a high-DPI image.

Text extraction snippet

import fitz

def extract_text(path: str) -> str:
    doc = fitz.open(path)
    text = "\n".join(page.get_text() for page in doc)
    doc.close()
    return text

Use this only when you are confident the PDF contains a real text layer.

Step 2: Install the toolchain

Use PyMuPDF (fitz) for rendering and requests for HTTP. Both are stable and pip-installable.

pip install pymupdf requests

Avoid heavyweight headless Chrome setups unless you need pixel-perfect CSS rendering. For PDFs, native PDF libraries are faster and deterministic.

Step 3: Render PDF pages to PNG

Open the document, iterate pages, and produce PNG bytes at a sensible DPI. 150 DPI is enough for most OCR-grade vision tasks and keeps payloads under control.

import fitz  # pymupdf

def render_pdf_pages(path: str, dpi: int = 150) -> list[bytes]:
    doc = fitz.open(path)
    png_pages = []
    for page in doc:
        pix = page.get_pixmap(dpi=dpi)
        png_pages.append(pix.tobytes("png"))
    doc.close()
    return png_pages

Watch memory on large PDFs. A 100-page document at 150 DPI yields ~100 MB of PNG bytes. Process page-by-page and send incrementally if needed. If you see truncated edges, bump DPI to 200 but monitor token cost—higher resolution means the model spends more visual tokens.

Step 4: Encode images as data URIs

Multimodal chat completions accept image_url with a data: URI. Base64-encode the PNG and prefix with the MIME type.

import base64

def to_data_uri(png_bytes: bytes) -> str:
    b64 = base64.b64encode(png_bytes).decode("ascii")
    return f"data:image/png;base64,{b64}"

Do not write temp files unless you plan to host them. Inline data URIs keep the request self-contained and sidestep URL signing. Be aware that base64 expands size by ~33%; factor that into request limits.

Step 5: Construct the request payload

The OpenAI chat completions schema is the de facto standard. A user message with a content array mixes text and images. When you send pdf documents multimodal api calls this way, the model sees each page as an image.

def build_payload(png_pages: list[bytes], model: str = "gpt-4o") -> dict:
    content = [{"type": "text", "text": "Extract structured data from these pages."}]
    for png in png_pages[:4]:  # limit to 4 pages per call
        content.append({
            "type": "image_url",
            "image_url": {"url": to_data_uri(png)}
        })
    return {
        "model": model,
        "messages": [
            {"role": "system", "content": "You are a precise document parser."},
            {"role": "user", "content": content}
        ],
        "max_tokens": 2048
    }

Keep page count per request inside the model’s context limit; 4–8 pages is safe for most 128k context windows. Place the text instruction before the images so the model anchors on the task before processing visuals.

Step 6: Post to an OpenAI-compatible endpoint

Use requests. The standard endpoint is https://api.openai.com/v1/chat/completions. If you route through n4n.ai, its OpenAI-compatible endpoint addresses 240+ models and automatically falls back when a provider is rate-limited or degraded, so the same payload works without rewriting client code.

import requests

API_KEY = "sk-..."  # your provider key
ENDPOINT = "https://api.openai.com/v1/chat/completions"

payload = build_payload(render_pdf_pages("invoice.pdf"))
resp = requests.post(
    ENDPOINT,
    headers={"Authorization": f"Bearer {API_KEY}"},
    json=payload,
    timeout=60
)
resp.raise_for_status()
data = resp.json()

Set a timeout. Vision requests with multiple images can take 20–40 seconds; a 60s timeout avoids premature hangs. For production, wrap the call in a retry loop with backoff on 429/5xx.

Step 7: Verify success and meter usage

A successful response carries choices[0].finish_reason == "stop" and a usage block. Check both before trusting the output.

assert data["choices"][0]["finish_reason"] == "stop", "Model truncated output"
usage = data["usage"]
print(f"Prompt tokens: {usage['prompt_tokens']}, completion: {usage['completion_tokens']}")
extracted = data["choices"][0]["message"]["content"]

If finish_reason is "length", raise max_tokens. If the API returns 429, implement exponential backoff. Gateways that provide per-token usage metering let you track cost per document without extra instrumentation. Log the usage block alongside the document hash for audit trails.

Step 8: Handle multi-page documents at scale

For a 50-page report, do not cram all pages into one call. Batch into chunks of 4–6 pages, or use a map-reduce pattern: extract per chunk, then summarize.

def process_large_pdf(path: str, chunk_size: int = 4):
    pages = render_pdf_pages(path)
    results = []
    for i in range(0, len(pages), chunk_size):
        chunk = pages[i:i+chunk_size]
        payload = build_payload(chunk)
        r = requests.post(ENDPOINT, headers={"Authorization": f"Bearer {API_KEY}"}, json=payload)
        r.raise_for_status()
        results.append(r.json()["choices"][0]["message"]["content"])
    return results

Persist intermediate results. A single failed chunk should not force re-rendering the whole PDF. Run chunks concurrently with a worker pool bounded to 5 threads to stay under rate limits.

Step 9: Forward cache and routing hints

Some gateways honor client routing directives and provider cache-control. If you call the same PDF repeatedly (e.g., a static contract), set a cache marker to avoid re-paying vision tokens.

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "X-Route-Model": "gpt-4o",
    "X-Cache-Control": "max-age=3600"
}
requests.post(ENDPOINT, headers=headers, json=payload)

n4n.ai forwards provider cache-control hints and honors client routing directives, which matters when you send pdf documents multimodal api traffic across multiple backends and want consistent caching behavior.

Verify your pipeline end to end

Create a one-page test PDF with known text. Render it, send the request, and assert that the returned content contains the expected string. If the model echoes the text, your image encoding and payload shape are correct. Then test a scanned image PDF to confirm DPI settings produce legible input.

The moment you can round-trip a known PDF through the API and get deterministic extraction, you have a production-ready document ingestion path. Tune DPI, page chunking, and model choice based on real corpus quality, not synthetic benchmarks.