Background agents: OpenAI's approach vs custom orchestration

Building background agents OpenAI style means leaning on the managed Assistants API (or the newer background mode in the Responses API) to handle threading, polling, and tool execution outside your request path. The alternative—custom orchestration—puts you in control of the task queue, state machine, and model calls, often across multiple providers. This head-to-head breaks down both approaches so you can pick the right foundation for long-running agent workflows.

What we’re comparing

A background agent is any LLM-driven process that runs decoupled from the initiating request: it accepts a task, does multi-step work, calls tools, and reports back later. OpenAI’s approach ships this as a managed service. You create an assistant, spawn a thread, post a message, and start a run; the platform executes the loop and exposes status via polling or webhooks.

Custom orchestration is you writing the loop. A worker pulls a job from a queue, calls a model (or several), executes tools, persists state, and repeats until done. You own the runtime, the retry logic, and the failure modes.

Capabilities

OpenAI’s managed runtime

The Assistants API gives you:

• Persistent threads that store conversation state server-side.
• Built-in tools: code interpreter, file search, and function calling.
• Automatic run loop: the model decides to call tools, the API waits for you to submit tool outputs, then continues.
• Webhook or polling-based status updates.

You are locked to OpenAI models (GPT-4o, o-series, etc.) and to the tool sandbox they provide. If you need to hit a PostgreSQL database directly or call an internal gRPC service, you must wrap it as a function the model can request and you must host the execution.

Custom orchestration

You define the tools, the model routing, and the state schema. Want to use Claude for summarization, a local Mixtral for classification, and GPT-4o for reasoning? Fine. Want to persist intermediate state in Postgres and emit events to Kafka? Fine.

A minimal custom worker using Celery and the OpenAI client:

from celery import Celery
from openai import OpenAI

app = Celery("agents", broker="redis://localhost")
client = OpenAI()  # or point base_url at a gateway

@app.task
def run_agent(task_id: str, user_msg: str):
    # load state from DB, call model, execute tools, loop
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": user_msg}],
        tools=[{"type": "function", "function": {"name": "get_order", "parameters": {...}}}],
    )
    # ... dispatch tool calls, persist, repeat

When building custom orchestration, routing across many models is simpler via an OpenAI-compatible endpoint like n4n.ai that addresses 240+ models with automatic fallback and per-token metering.

Failure handling

OpenAI runs surface a terminal failed or expired status; you must inspect last_error and decide whether to retry by creating a new run. There is no built-in saga or compensation. Custom orchestration lets you implement exponential backoff, dead-letter queues, and human-review pauses directly in your worker code.

Price and cost model

OpenAI does not charge a separate platform fee for the Assistants API. You pay per token for model inference, and the thread storage is free but counts against your context window. The catch: each run step resends the relevant thread context to the model, so long threads can quietly multiply token cost. Tool calls and code interpreter sessions also bill tokens for generated code and outputs.

Custom orchestration has the same per-token model cost, but you add infrastructure: a queue, a worker pool, a database. Those are typically cheap at low volume (a single Redis + Postgres instance) and scale predictably. The upside is you can trim context, cache embeddings, or route to cheaper models for trivial steps—controls the managed API does not expose. If you use a gateway that forwards provider cache-control hints, you can also exploit provider-side prompt caching to cut cost.

Latency and throughput

OpenAI’s run loop is asynchronous by design. After creating a run, you poll run.status or wait for a webhook. Expect seconds to minutes depending on tool steps and platform load. Streaming tokens is not available inside a background run; you get the final aggregated message. You do not control concurrency beyond your account rate limits; if OpenAI is degraded, your agents stall.

Custom orchestration lets you tune worker concurrency, batch jobs, and backoff. You can run hundreds of agents in parallel on your own Kubernetes cluster. The trade-off is that you must implement rate-limit handling for upstream model APIs. If you use a gateway with automatic fallback when a provider is rate-limited, you offload some of that pain.

Ergonomics

The Assistants API is undeniably faster to scaffold. A few REST calls and you have a persistent agent. Debugging, however, is opaque: you see run steps but not the exact prompt sent, and replaying a run requires reconstructing thread state.

Custom orchestration is more code but gives you full observability. You can log every model request, every tool input/output, and trace latency per step with OpenTelemetry. For teams already running background jobs (Temporal, BullMQ, Celery), the agent is just another worker.

Example: creating an assistant and polling with the OpenAI SDK.

from openai import OpenAI
client = OpenAI()

assistant = client.beta.assistants.create(
    model="gpt-4o",
    tools=[{"type": "code_interpreter"}],
)
thread = client.beta.threads.create()
client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content="Generate a sales chart from Q3 data",
)
run = client.beta.threads.runs.create(
    thread_id=thread.id, assistant_id=assistant.id
)
# poll run.status until completed

Contrast with a custom loop where the polling is your own queue status and you can inspect the exact payload.

Ecosystem and integration

OpenAI’s approach integrates natively with its model roadmap: new models appear in the API without you changing code. Its file search and code interpreter are turnkey. But it sits outside your existing CI/CD and observability stack unless you build bridges.

Custom orchestration plugs into whatever you already use. Need to trigger an agent from a GitHub Action? Publish to the same queue. Need to enforce SPIFFE auth on tool calls? Your code, your rules. You can also mix in non-OpenAI models or self-hosted weights, and honor client routing directives to pin specific providers.

Limits and constraints

Assistants API limits:

• Maximum context window per model (e.g., 128k for GPT-4o) applies to the thread.
• Run timeouts exist; very long tasks may need chunking.
• No custom runtime; you cannot run a long-lived process inside their sandbox beyond tool calls.

Custom orchestration limits:

• You are responsible for idempotency, dead-letter queues, and state migration.
• Model provider rate limits still apply; you must handle 429s.
• More surface area for bugs in the orchestration layer itself.

Head-to-head summary

Which to choose

Choose OpenAI’s background agents if:

• You are prototyping or building a single-tenant internal tool.
• You only need OpenAI models and the built-in code interpreter/file search.
• Your team has no existing job infrastructure and wants to ship in a day.
• The tasks fit inside a single thread context and complete within run timeout.

Choose custom orchestration if:

• You need to combine multiple model providers or route by cost/latency.
• Compliance requires data to stay in your VPC and tool calls to be audited.
• You already run a distributed worker system and want agents as first-class jobs.
• Long-running workflows exceed OpenAI’s run limits or need human-in-the-loop pauses longer than the API supports.
• You want to apply provider prompt caching or fine-grained token metering.

Hybrid pattern: Start with OpenAI’s managed API to validate the agent logic. Once the workflow is stable and you hit cost or scale walls, extract the loop into a custom worker that speaks the same tool schema. This avoids over-engineering early while leaving a clean migration path.

The decision is less about which is “better” and more about who owns the runtime. If you want the platform to own it, use background agents openai provides. If you need to own it, build the loop.