From openai.ChatCompletion to LlamaIndex: what changes

Most teams start with the raw OpenAI SDK because it’s simple: one import, one function call, done. But as soon as you need retrieval, multi-step reasoning, or observability, the boilerplate explodes. This guide walks through an openai chatcompletion to llamaindex migration with working code at each step, so you can move incrementally without rewriting your entire stack.

Why migrate at all

The raw SDK gives you chat completions. LlamaIndex gives you a composable pipeline: data connectors, index structures, query engines, and agent loops — all with pluggable LLMs. If you’re building RAG, agents, or any workflow that chains multiple LLM calls, the framework pays for itself in reduced glue code. The tradeoff is abstraction leakage: you’ll debug framework internals occasionally, and the API surface is larger.

1. Swap the client, keep the call shape

LlamaIndex’s OpenAI class wraps the same HTTP client. Your first migration step is dropping in the replacement without changing prompt logic.

# before: raw openai
from openai import OpenAI

client = OpenAI(api_key="sk-...")
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this: ..."}],
)
print(resp.choices[0].message.content)

# after: llamaindex openai wrapper
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini", api_key="sk-...")
resp = llm.complete("Summarize this: ...")
print(resp.text)

Pitfall: llm.complete() returns a CompletionResponse object, not a dict. Access .text or .raw for the provider payload. If you need the full OpenAI response (token counts, finish reason), use llm.chat(messages) which returns a ChatResponse with .message and .raw.

2. Migrate message-based chat

Most production code uses chat.completions.create with message arrays. LlamaIndex provides ChatMessage and a chat() method that mirrors the OpenAI shape.

from llama_index.core.llms import ChatMessage, MessageRole
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini")

messages = [
    ChatMessage(role=MessageRole.SYSTEM, content="You are a terse summarizer."),
    ChatMessage(role=MessageRole.USER, content="Summarize: ..."),
]

resp = llm.chat(messages)
print(resp.message.content)
print(resp.raw.usage)  # token usage if you need it

Tradeoff: LlamaIndex’s ChatMessage is a thin dataclass. It serializes to the same JSON the OpenAI API expects, so you can log or persist messages without translation.

3. Enable streaming the right way

Raw SDK streaming returns an iterator of chunks. LlamaIndex exposes stream_chat and stream_complete with the same pattern, but the chunk type differs.

# raw openai streaming
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a haiku"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

# llamaindex streaming
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini")
stream = llm.stream_chat([
    ChatMessage(role=MessageRole.USER, content="Write a haiku")
])
for chunk in stream:
    # chunk.delta is a string, not a nested object
    print(chunk.delta, end="", flush=True)

Pitfall: In LlamaIndex ≤0.10, stream_chat yielded ChatResponse objects with .delta as a string. In ≥0.11, it yields ChatResponseAsyncGen chunks — check your version. The safest pattern is for chunk in stream: print(chunk.delta or "", end="") which handles both.

4. Tool calling: from manual JSON to Pydantic

This is where the migration pays off. Raw SDK tool calling requires you to construct JSON schemas by hand, parse tool_calls from the response, execute functions, and feed results back. LlamaIndex’s FunctionTool and OpenAIAgent automate the loop.

# raw openai tool calling (abbreviated)
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
# ... call, parse tool_calls, execute, call again with tool results ...

# llamaindex: define tool as a python function
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI
from llama_index.core.agent import FunctionCallingAgentWorker
from llama_index.core.agent import AgentRunner

def get_weather(city: str) -> str:
    # real impl would call an API
    return f"{city}: 72°F, sunny"

weather_tool = FunctionTool.from_defaults(fn=get_weather)

llm = OpenAI(model="gpt-4o-mini")
worker = FunctionCallingAgentWorker.from_tools(
    [weather_tool], llm=llm, verbose=True
)
agent = AgentRunner(worker)

response = agent.chat("What's the weather in Tokyo?")
print(response)

What changed: You write a typed Python function. LlamaIndex generates the JSON schema, handles the tool-call/response loop, and returns the final answer. The agent also manages conversation history automatically.

Pitfall: FunctionCallingAgentWorker only works with models that support the OpenAI function-calling API (GPT-4, GPT-3.5-turbo, and compatible endpoints). For other models, use ReActAgentWorker which prompts the model to emit tool calls as text.

5. Structured output with Pydantic

If you’re parsing JSON from model output manually, stop. LlamaIndex’s PydanticOutputParser and structured_predict handle validation and retries.

from pydantic import BaseModel, Field
from llama_index.core.output_parsers import PydanticOutputParser
from llama_index.llms.openai import OpenAI
from llama_index.core.program import LLMTextCompletionProgram

class Extraction(BaseModel):
    company: str = Field(description="Company name")
    revenue_usd: float = Field(description="Annual revenue in USD")
    fiscal_year: int

parser = PydanticOutputParser(output_cls=Extraction)
llm = OpenAI(model="gpt-4o-mini")

program = LLMTextCompletionProgram.from_defaults(
    output_parser=parser,
    llm=llm,
    prompt_template_str=(
        "Extract company info from: {text}\n"
        "Return ONLY valid JSON matching the schema."
    ),
    verbose=True,
)

result = program(text="Acme Corp reported $1.2B revenue in FY2023.")
print(result.company, result.revenue_usd, result.fiscal_year)
# Acme Corp 1200000000.0 2023

Tradeoff: This adds a validation round-trip on failure (default 3 retries). For high-volume extraction, consider OpenAIPydanticProgram which uses the provider’s native response_format parameter — faster, but requires GPT-4o-2024-08-06 or later.

6. Plug in your own endpoint (OpenAI-compatible)

If you route through a gateway like n4n.ai for fallback, caching, or multi-provider access, you only need to change the base_url and api_key. LlamaIndex’s OpenAI class accepts both.

from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="gpt-4o-mini",           # or any model your gateway serves
    api_key="n4n_sk_...",          # gateway key
    base_url="https://api.n4n.ai/v1",  # gateway endpoint
    # optional: pass through routing hints
    additional_kwargs={"extra_headers": {"x-n4n-model": "prefer:claude-3.5-sonnet"}},
)

The gateway handles provider fallback and returns standard OpenAI-shaped responses, so the rest of your LlamaIndex code stays unchanged.

7. Observability: callbacks instead of manual logging

Raw SDK users often wrap calls with custom logging. LlamaIndex has a callback system that fires on every LLM call, embedding, and tool execution.

from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler

# global debug handler prints timing, token counts, prompts
debug = LlamaDebugHandler(print_trace_on_end=True)
Settings.callback_manager = CallbackManager([debug])

# now every llm.chat(), embed_model.get_text_embedding(), etc. is logged
response = llm.chat([ChatMessage(role=MessageRole.USER, content="Hi")])
# prints: LLM call took 1.2s, 45 prompt tokens, 12 completion tokens

For production, swap LlamaDebugHandler for a custom handler that ships to your observability stack (Datadog, Honeycomb, etc.). The callback receives CBEventType.LLM events with payload containing the serialized request/response.

8. Incremental adoption: keep raw calls where they work

You don’t need a big-bang rewrite. A common pattern: use LlamaIndex for RAG/agent workflows, keep raw SDK for simple chat endpoints.

# simple_chat.py — stays on raw sdk
from openai import OpenAI
from fastapi import FastAPI

app = FastAPI()
client = OpenAI()

@app.post("/chat")
async def chat(msg: str):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": msg}],
        stream=True,
    )
    return StreamingResponse(resp, media_type="text/event-stream")

# rag_pipeline.py — uses llamaindex
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(llm=OpenAI(model="gpt-4o-mini"))

response = query_engine.query("What's our refund policy?")

Both can share the same gateway endpoint and API key. Migrate piece by piece.

9. Common migration failures

10. What you lose and gain

Lose: Direct control over every HTTP request. You can’t easily inject custom headers per-call without additional_kwargs, and retry logic is internal.

Gain: Composability. The same llm instance works in VectorStoreIndex, SummaryIndex, KnowledgeGraphIndex, FunctionCallingAgent, ReActAgent, and custom pipelines. You swap the LLM once (e.g., to a local model via Ollama or LlamaCPP) and the entire stack adapts.

Verdict: If your product is a thin chat wrapper, stay on the raw SDK. If you’re building anything that retrieves, reasons over tools, or chains prompts — migrate. The boilerplate you delete outweighs the abstraction cost.