Vercel AI SDK multi-modal inputs: images and PDFs tutorial

The vercel ai sdk multi-modal images pdfs workflow is simpler than most teams assume, but the documentation scatters the pieces across provider packages. This tutorial builds a runnable Node script that pushes an image and a PDF into a single model call using the Vercel AI SDK, with checkpoints so you can verify each step. We’ll use Anthropic’s Claude because it natively accepts both input types through the SDK’s standard message shape.

Prerequisites

• Node.js 18 or newer (uses global fetch and Blob).
• An Anthropic API key (or any vision-capable endpoint). Set it as ANTHROPIC_API_KEY in your environment.
• A local image (./receipt.png) and a PDF (./contract.pdf) to test with.
• Familiarity with TypeScript and ES modules.

Initialize a project and install the SDK:

mkdir multimodal-demo && cd multimodal-demo
npm init -y
npm install ai @ai-sdk/anthropic

Set "type": "module" in package.json so we can use top-level await.

Sending an image

The SDK represents multi-part user messages as an array of content blocks. For an image, use { type: 'image', image: Uint8Array }. The image field accepts a Uint8Array, Buffer, or URL string.

Create image.ts:

import { generateText } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';
import { readFile } from 'node:fs/promises';

const image = await readFile('./receipt.png');

const { text } = await generateText({
  model: anthropic('claude-3-5-sonnet-20240620'),
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'What is the total amount on this receipt?' },
        { type: 'image', image },
      ],
    },
  ],
});

console.log(text);

Run it:

ANTHROPIC_API_KEY=sk-... node image.ts

Expected output (varies by image):

The total amount on the receipt is $42.17, dated 2024-05-12.

If you get a 413, the image is likely too large; downscale or compress before reading. Claude accepts up to 20MB per image, but smaller is faster and cheaper.

Sending a PDF

PDFs are not a first-class content part in the CoreMessage type, but the Anthropic provider maps experimental_attachments to the model’s document format. Pass the file as a data URL with the correct MIME type.

Create pdf.ts:

import { generateText } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';
import { readFile } from 'node:fs/promises';

const pdf = await readFile('./contract.pdf');
const pdfDataUrl = `data:application/pdf;base64,${pdf.toString('base64')}`;

const { text } = await generateText({
  model: anthropic('claude-3-5-sonnet-20240620'),
  messages: [
    {
      role: 'user',
      content: 'List the termination clauses in this document.',
      experimental_attachments: [
        { name: 'contract.pdf', contentType: 'application/pdf', url: pdfDataUrl },
      ],
    },
  ],
});

console.log(text);

Expected output:

1. Either party may terminate with 30 days written notice.
2. Breach of payment terms allows immediate termination.
3. Force majeure events suspend obligations for 60 days.

The experimental_attachments array is provider-specific. OpenAI’s vision models ignore PDFs; you would need to extract text first. Anthropic treats the attachment as a native PDF block, preserving layout.

Combining image and PDF in one request

You can mix a text prompt, an inline image, and a PDF attachment in the same user turn. The model receives them in order.

import { generateText } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';
import { readFile } from 'node:fs/promises';

const image = await readFile('./diagram.png');
const pdf = await readFile('./spec.pdf');
const pdfDataUrl = `data:application/pdf;base64,${pdf.toString('base64')}`;

const { text } = await generateText({
  model: anthropic('claude-3-5-sonnet-20240620'),
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Does the diagram match the system spec? Note discrepancies.' },
        { type: 'image', image },
      ],
      experimental_attachments: [
        { name: 'spec.pdf', contentType: 'application/pdf', url: pdfDataUrl },
      ],
    },
  ],
});

console.log(text);

This is the core of the vercel ai sdk multi-modal images pdfs pattern: content blocks for inline media, attachments for documents.

Streaming responses

For production, stream to avoid blocking on large documents. Swap generateText for streamText and iterate.

import { streamText } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';
import { readFile } from 'node:fs/promises';

const pdf = await readFile('./contract.pdf');
const pdfDataUrl = `data:application/pdf;base64,${pdf.toString('base64')}`;

const result = await streamText({
  model: anthropic('claude-3-5-sonnet-20240620'),
  messages: [
    {
      role: 'user',
      content: 'Summarize this PDF in three sentences.',
      experimental_attachments: [
        { name: 'contract.pdf', contentType: 'application/pdf', url: pdfDataUrl },
      ],
    },
  ],
});

for await (const delta of result.textStream) {
  process.stdout.write(delta);
}

You get token-by-token output. Handle result.error in a try/catch; provider rate limits surface as standard errors.

Using remote image URLs

Instead of reading from disk, pass a URL string directly to the image field. The SDK fetches it during the call.

const { text } = await generateText({
  model: anthropic('claude-3-5-sonnet-20240620'),
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Describe this screenshot.' },
        { type: 'image', image: 'https://example.com/dashboard.png' },
      ],
    },
  ],
});

Use this when your assets live in object storage. Ensure the URL is publicly reachable or carries a signed token.

Extracting PDF text when the model can’t read PDFs

If you must target a model without native PDF support, extract text locally and send it as a text block. Install a parser:

npm install pdf-parse

import pdfParse from 'pdf-parse';
import { readFile } from 'node:fs/promises';
import { generateText } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';

const buf = await readFile('./contract.pdf');
const { text: pdfText } = await pdfParse(buf);

const { text } = await generateText({
  model: anthropic('claude-3-5-sonnet-20240620'),
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Summarize the following document:' },
        { type: 'text', text: pdfText.slice(0, 12000) },
      ],
    },
  ],
});

This loses visual layout but works on any text model. Truncate to fit context limits.

Routing through a single gateway

If you don’t want to pin your code to one provider package, you can use the OpenAI-compatible provider against a gateway. n4n.ai exposes one OpenAI-compatible endpoint that fronts 240+ models and automatically falls back when a provider is rate-limited or degraded, so the same image content part works against GPT-4o or Claude without swapping imports. The SDK call looks like:

import { openai } from '@ai-sdk/openai';
const model = openai('claude-3-5-sonnet-20240620', { baseURL: process.env.GATEWAY_URL });

The message shape for images and attachments stays identical; the gateway forwards provider cache-control hints and meters per token.

Gotchas and limits

• Token counting: PDFs are tokenized from rendered pages, not raw bytes. A 10-page PDF can be 3k–8k tokens. Check context windows before attaching.
• Base64 overhead: Data URLs add ~33% size. For images, prefer a remote URL if the provider supports it natively.
• Experimental API: experimental_attachments may change across SDK minors. Pin exact versions in package.json and read release notes.
• Provider mismatch: OpenAI ignores experimental_attachments for PDFs. If you switch models, verify the provider maps your inputs or you’ll silently send empty context.
• Error shapes: Provider errors in the SDK throw AI_APICallError. Catch and inspect error.statusCode to implement backoff.

Final checklist

• Image sent as { type: 'image', image } content block.
• PDF sent as experimental_attachments with application/pdf MIME.
• Used generateText for tests, streamText for UX.
• Verified output before chaining into downstream logic.
• Considered gateway routing to avoid provider lock-in.

The vercel ai sdk multi-modal images pdfs path is now a repeatable pattern: content array for vision, attachments for docs, one model call. Build from there.