Long-document agents stop being a retrieval problem when the gemini 3 context window lets you pipe entire PDFs, scans, and chat history into a single prompt. But a large context isn’t a free lunch: you still have to manage token cost, cache boundaries, and multimodal serialization. This guide gives an ordered path to build production document agents on Gemini 3 without drowning in latency or bill shock.
Why the gemini 3 context window changes agent design
Skip the vector database for many workflows. When the model can see the whole corpus, you replace approximate search with exact attention. That eliminates chunk-miss errors and re-ranking complexity that plague standard RAG stacks.
The tradeoff is real: every token is processed on each turn unless you use cache control. Multimodal inputs (images, layouts) inflate token counts faster than plain text. Design for that from day one, or your first invoice will be the only evaluation you get.
Step 1: Inventory the document topology
Before writing a line of agent code, map what you’re feeding. Separate native text, OCR-needed scans, tables, and embedded images. The gemini 3 context window handles mixed modalities, but each has different tokenization cost and failure modes.
Token math for mixed modalities
A text page is cheap. A scanned page sent as an image is not. Probe a single page to learn the multiplier:
import tiktoken, base64
enc = tiktoken.get_encoding("cl100k_base")
text = open("page1.txt").read()
text_tokens = len(enc.encode(text))
# Image token cost is model-specific; log it from the usage field
print(f"text tokens: {text_tokens}")Do not assume the PDF text layer matches the visual layout. If a page is a scan, you must send image parts, and the gemini 3 context window will consume vision tokens per tile.
Step 2: Decide inline vs retrieved context
Not everything belongs in the prompt. Use this rule: if the agent needs cross-document synthesis across >50 files, use a lightweight index and pull only relevant passages. For a single large doc (up to the gemini 3 context window limit), inline it.
When to still use RAG
- Corpus exceeds a few million tokens total.
- Per-query latency budget is under 2 seconds.
- You need source citations across thousands of docs.
Hybrid pattern works best:
- System prompt: agent instructions + output schema.
- Static context: whole document with cache control.
- Dynamic context: user query + tool results appended per turn.
Step 3: Serialize documents for the gemini 3 context window
Pack the document with explicit delimiters. Models attend better with structure markers than raw concatenation. Losing page order or mixing image and text without labels degrades reasoning.
{
"document_id": "lease-2024",
"pages": [
{"page": 1, "text": "...", "images": ["data:image/png;base64,..."]},
{"page": 2, "text": "..."}
]
}When calling an OpenAI-compatible gateway, send multimodal parts in order:
from openai import OpenAI
import json
client = OpenAI(base_url="https://api.openai-compatible.example/v1", api_key="sk-...")
doc_payload = json.load(open("doc.json"))
resp = client.chat.completions.create(
model="gemini-3-pro",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Analyze the following document:"},
{"type": "text", "text": json.dumps(doc_payload)}
]
}]
)For true image parts, use the part type your endpoint expects. Keep page order intact; shuffling kills layout reasoning. The gemini 3 context window is wide, but it is not a reordering buffer.
Step 4: Set cache boundaries to control cost
The gemini 3 context window makes reprocessing expensive if you resend the document every turn. Use ephemeral cache control on the static document block.
curl https://api.openai-compatible.example/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-d '{
"model": "gemini-3-pro",
"messages": [
{"role": "system", "content": "You are a lease analyzer."},
{"role": "user", "content": "DOC_START", "cache_control": {"type": "ephemeral"}},
{"role": "user", "content": "<full document>"},
{"role": "user", "content": "DOC_END", "cache_control": {"type": "ephemeral"}}
]
}'Cache hits skip recompute of the marked prefix. Without this, a 10-turn conversation re-pays for the document 10 times. Any change to the cached prefix—even trailing whitespace—invalidates it.
Cache control nuances
- Mark only the immutable document, not the system instruction if you tweak it per call.
- Ephemeral caches typically live minutes; don’t assume cross-session reuse.
- If your gateway forwards provider cache-control hints, the same header shape works end to end.
Step 5: Wire routing and fallback for production
Providers degrade. If you pin Gemini 3 and it rate-limits, your agent stalls. A gateway that honors client routing directives and forwards provider cache-control hints lets you prefer Gemini 3 but automatically fall back when degraded.
curl https://api.n4n.ai/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "X-Route-Prefer: gemini-3-pro" \
-d '{"model":"gemini-3-pro","messages":[{"role":"user","content":"Summarize"}]}'The request stays OpenAI-compatible; the gateway tries Gemini 3, then shifts to an available equivalent if needed. Per-token metering still applies, so cost stays observable. This is the only place where fronting with a gateway earns its keep for document agents: you keep the large context strategy and lose the single-provider fragility.
Step 6: Implement a bounded agent loop
Long-doc agents should not call the model in an open loop. Set max steps, and only append new evidence per step. Keep the cached document message reference stable across steps so the prefix stays cached.
MAX_STEPS = 5
history = [system_msg, cached_doc_msg]
for step in range(MAX_STEPS):
user_task = get_next_subtask()
history.append({"role": "user", "content": user_task})
resp = client.chat.completions.create(model="gemini-3-pro", messages=history)
history.append({"role": "assistant", "content": resp.choices[0].message.content})
if "FINAL" in resp.choices[0].message.content:
breakStream the response. First-token latency grows with prefix size; a progress UI is mandatory for documents near the gemini 3 context window ceiling.
Common pitfalls and tradeoffs
Context rot: Even with a huge window, attention degrades on needle-in-haystack tasks beyond certain depths. Put critical instructions at start and end of the prompt.
Token inflation: Multimodal pages can blow past expected token counts. Probe one page before sending 500.
Cache invalidation: Any edit to the cached prefix invalidates it. Treat the document block as immutable; mutate only the trailing messages.
Latency: First token lags with large prefixes. Budget for it; don’t promise synchronous responses.
Over-reliance on inline: For 10,000 documents, inline everything is absurd. Use the gemini 3 context window for depth, not breadth.
Evaluation blind spots: Exact-match metrics fail when the model reads the whole doc but mis-anchors a clause. Build golden-answer tests per document type.
Minimal reference implementation
Combine the pieces: serialize, cache, route, loop.
import json, os
from openai import OpenAI
client = OpenAI(base_url=os.environ["OPENAI_BASE_URL"], api_key=os.environ["KEY"])
doc = json.load(open("doc.json"))
cached_prefix = [
{"role": "system", "content": "Doc agent."},
{"role": "user", "content": "DOC_START", "cache_control": {"type": "ephemeral"}},
{"role": "user", "content": json.dumps(doc)},
{"role": "user", "content": "DOC_END", "cache_control": {"type": "ephemeral"}}
]
history = cached_prefix.copy()
for q in ["Summarize", "List parties", "Flag risks"]:
history.append({"role": "user", "content": q})
r = client.chat.completions.create(model="gemini-3-pro", messages=history)
history.append({"role": "assistant", "content": r.choices[0].message.content})Swap OPENAI_BASE_URL to a gateway that supports the routing header if you need fallback. The gemini 3 context window does the heavy lifting; your job is to keep the payload disciplined, cached, and bounded.