Batching LLM calls to cut agent inference costs

Agents that fire dozens of independent completions per task burn money on synchronous round-trips. Batching LLM calls cost drops sharply when you collapse those requests into a single provider batch, trading latency for a steep per-token discount. This tutorial builds a small batch dispatcher that queues independent agent subtasks and submits them through an OpenAI-compatible batch endpoint.

Prerequisites

Python 3.10 or newer
openai Python SDK >= 1.0 (pip install openai)
An API key for an OpenAI-compatible inference service, exported as LLM_API_KEY
A BASE_URL for your provider (default OpenAI shown; swap for your gateway)

No prior batch experience required. You should understand that batch endpoints process requests asynchronously, usually within a 24-hour window, and return results in a file. If you already run an agent loop with client.chat.completions.create calls in a loop, you have the raw material for batching.

Why batching fits agent workloads

Most agent frameworks execute a plan as a tree of steps. Many leaves are independent: summarizing retrieved documents, classifying intents, or extracting structured fields. Firing those as synchronous calls serializes network latency and forfeits batch pricing. Batching LLM calls cost is lower because providers amortize scheduling overhead and offer reduced per-token rates for deferred execution.

The tradeoff is simple: you cannot use the result for at least minutes, often hours. That rules out interactive chat, but not offline evaluation, bulk ingestion, or nightly report agents.

Step 1: Identify batchable agent subtasks

A subtask is batchable only if it does not depend on the output of another subtask in the same batch. Typical examples: per-customer summarization, bulk classification, parallel tool-result parsing.

tasks = [
    "Summarize the following support ticket: 'Login fails on Safari'",
    "Classify this request as billing or technical: 'Refund not processed'",
    "Extract key entities from: 'Order #123 shipped to Berlin'",
]

If your agent loop generates these during execution, append them to a list instead of calling the model immediately. In a real agent, you might intercept completion calls tagged with batchable=True in your orchestration layer.

Step 2: Build batch request objects

The OpenAI batch API accepts a JSONL file where each line is a request with a custom_id, method, url, and body. The body mirrors a normal chat completion call.

import json
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("BASE_URL", "https://api.openai.com/v1")
)

model = "gpt-4o-mini"  # or any supported chat model
batch_requests = []
for i, prompt in enumerate(tasks):
    batch_requests.append({
        "custom_id": f"task-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "max_tokens": 200
        }
    })

with open("batch_input.jsonl", "w") as f:
    for req in batch_requests:
        f.write(json.dumps(req) + "\n")

This file is the only artifact you submit. Keep custom_id stable; it maps responses back to your tasks. Avoid spaces or special characters in the ID to simplify later parsing.

Step 3: Upload and create the batch

Upload the JSONL as a file with purpose="batch", then create a batch with a completion window. The 24h window is standard for discounted processing.

file_obj = client.files.create(
    file=open("batch_input.jsonl", "rb"),
    purpose="batch"
)

batch = client.batches.create(
    input_file_id=file_obj.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

print(f"Batch {batch.id} created, status: {batch.status}")

Expected output at this checkpoint:

Batch batch_abc123xyz created, status: pending

The batch is now queued. No tokens are spent until the provider begins processing. The returned batch object also exposes request_counts which starts at zero and increments as the batch progresses.

Step 4: Poll for completion and read results

Batch processing is not instant. Poll at a coarse interval—thirty seconds is polite for a job that may take hours. When status is completed, download the output file.

import time

while True:
    status = client.batches.retrieve(batch.id)
    if status.status in ("completed", "failed", "expired"):
        break
    time.sleep(30)

if status.status != "completed":
    raise RuntimeError(f"Batch ended with status {status.status}")

output = client.files.content(status.output_file_id)
for line in output.text.splitlines():
    res = json.loads(line)
    cid = res["custom_id"]
    content = res["response"]["body"]["choices"][0]["message"]["content"]
    print(f"{cid}: {content[:60]}...")

Sample output after completion:

task-0: Login issue on Safari likely due to cookie settings...
task-1: technical
task-2: entities: order_id=123, city=Berlin

Each line is a full completion. Failed individual requests appear with an error field rather than crashing the batch.

Step 5: Measure and reduce batching LLM calls cost

Batching LLM calls cost is driven by two levers: the provider’s batch discount and model selection. OpenAI’s batch endpoint historically halves per-token pricing versus synchronous calls; other gateways expose similar structures.

To verify savings, meter token usage. The response bodies include usage objects:

total_tokens = 0
for line in output.text.splitlines():
    res = json.loads(line)
    total_tokens += res["response"]["body"]["usage"]["total_tokens"]
print(f"Total tokens across batch: {total_tokens}")

An OpenAI-compatible gateway like n4n.ai forwards provider cache-control hints and meters per-token usage across 240+ models, so you can confirm batch discounts and apply automatic fallback if a provider is rate-limited mid-batch.

When evaluating batching LLM calls cost, factor in the latency tradeoff: a 24h window is useless for interactive agents but ideal for nightly report generation, bulk eval, or offline dataset enrichment.

Below is a qualitative comparison:

Dimension	Synchronous calls	Batch submission
Latency	Seconds	Hours (deferred)
Pricing	Standard rate	Reduced rate
Use case	Real-time agents	Bulk, offline
Failure	Immediate raise	Per-line errors

Step 6: Make your agent batch-aware

Wrap the submission logic in a class that collects tasks during an agent run and flushes at the end.

class BatchAgent:
    def __init__(self, client, model):
        self.client = client
        self.model = model
        self.pending = []

    def defer(self, prompt: str):
        self.pending.append(prompt)

    def flush(self):
        if not self.pending:
            return {}
        reqs = []
        for i, p in enumerate(self.pending):
            reqs.append({
                "custom_id": f"t-{len(self.pending)}-{i}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {"model": self.model,
                         "messages": [{"role": "user", "content": p}],
                         "max_tokens": 200}
            })
        with open("tmp_batch.jsonl", "w") as f:
            for r in reqs:
                f.write(json.dumps(r) + "\n")
        fobj = self.client.files.create(file=open("tmp_batch.jsonl", "rb"), purpose="batch")
        batch = self.client.batches.create(
            input_file_id=fobj.id, endpoint="/v1/chat/completions", completion_window="24h"
        )
        self.pending.clear()
        return {"batch_id": batch.id}

Call defer inside your agent loop, flush when the task graph hits a sync point that can wait. For long-running agents, a background thread can flush every N tasks or M seconds.

Step 7: Handling per-line errors

A completed batch may still contain failed requests. Inspect each line:

failed = []
for line in output.text.splitlines():
    res = json.loads(line)
    if "error" in res:
        failed.append(res["custom_id"])
    else:
        # process res["response"]["body"]
        pass
if failed:
    print(f"Retrying {len(failed)} failed tasks synchronously")

Because batch pricing only charges successful tokens on some providers, isolated failures are cheap to replay via synchronous calls.

Pitfalls and hard limits

Hidden dependencies: If task B uses task A’s output, batching introduces a correctness bug. Static analysis of your prompt graph is required.
Partial failure: A batch can complete with some lines errored. Always check response vs error per line.
File size caps: Providers limit batch input file size (often 100 MB). Shard large jobs.
Model availability: Not every model supports batch. Confirm before submitting.

Batching LLM calls cost is a straightforward win for offline agent workloads, but it demands discipline in task isolation. Build the dispatcher once, instrument token counts, and let your gateway’s metering show the delta.

Extending to multi-provider routing

If you submit batches through a gateway that honors client routing directives, you can pin each body to a different model or provider by adding a header hint in the request URL query or via gateway-specific extension. Keep the batch JSONL schema valid for the base endpoint; routing is resolved server-side. This lets you mix a cheap summarizer with a stronger classifier in one batch without code forks.

That’s the full loop: collect, submit, poll, parse. The next time your agent spins up a thousand independent completions, ship them as one batch and watch the invoice shrink.