Which AI framework fits a solo developer's side project

You’re building a side project with LLMs. You have limited time, zero budget for vendor lock-in, and you need something that won’t require a rewrite when you inevitably change models or providers. The framework you pick today determines how much pain you feel tomorrow. Here’s how to choose without overthinking it.

Start with the honest question

Before evaluating frameworks, answer this: what is the actual LLM workload?

• Chat with tools: User sends messages, model calls functions, returns results. Most side projects.
• RAG over documents: Ingest PDFs, chunk, embed, retrieve, synthesize. Knowledge bases, docs chat.
• Structured extraction: Unstructured input → validated JSON/schema. Invoices, resumes, logs.
• Multi-step agents: Planning, reflection, loops, memory. Research assistants, coding agents.
• Streaming UI: Token-by-token to the browser, optimistic updates, abort handling. Chat interfaces.

If you can’t name the pattern, you don’t need a framework yet. Call the API directly.

The decision tree

Is this a Next.js/React app with streaming chat UI?
  └── Yes → Vercel AI SDK (useChat, useCompletion)
  
Do you need RAG with minimal custom logic?
  └── Yes → LlamaIndex (data connectors + query engines)
  
Do you need complex chains, multiple tools, or agent loops?
  └── Yes → LangChain (LCEL + LangGraph)
  
Everything else?
  └── Raw HTTP + Pydantic/Zod schemas

This covers 90% of side projects. The remaining 10% need custom orchestration — build it yourself.

Option 1: Vercel AI SDK — best for streaming chat UIs

If your side project is a web app with a chat interface, this is the default choice. It handles the hard parts: streaming tokens to the client, managing conversation state, aborting in-flight requests, and tool calling with type safety.

// app/api/chat/route.ts
import { streamText, tool } from 'ai';
import { openai } from '@ai-sdk/openai';
import { z } from 'zod';

export async function POST(req: Request) {
  const { messages } = await req.json();
  
  const result = await streamText({
    model: openai('gpt-4o-mini'),
    messages,
    tools: {
      getWeather: tool({
        parameters: z.object({
          location: z.string().describe('City name'),
        }),
        execute: async ({ location }) => {
          const res = await fetch(`https://api.weather.gov/points/${location}`);
          return res.json();
        },
      }),
    },
    maxSteps: 5, // enables multi-step tool calling
  });
  
  return result.toDataStreamResponse();
}

// components/Chat.tsx
'use client';
import { useChat } from '@ai-sdk/react';

export function Chat() {
  const { messages, input, handleInputChange, handleSubmit, isLoading } = useChat();
  
  return (
    <div>
      {messages.map(m => (
        <div key={m.id} className={m.role}>
          {m.content}
        </div>
      ))}
      <form onSubmit={handleSubmit}>
        <input value={input} onChange={handleInputChange} disabled={isLoading} />
        <button type="submit" disabled={isLoading}>Send</button>
      </form>
    </div>
  );
}

Pitfalls:

• Tied to Vercel/Next.js ecosystem (though ai package works anywhere)
• Tool calling abstraction leaks when providers differ in schema support
• No built-in memory or persistence — you own the conversation store

When to skip: Non-React frontends, CLI tools, background workers, or when you need complex pre/post-processing pipelines.

Option 2: LlamaIndex — best for RAG that just works

LlamaIndex excels at the ingestion → index → query pipeline. Its data connectors (LlamaHub) handle PDFs, Notion, GitHub, SQL, and 100+ sources. The query engine abstraction lets you swap retrieval strategies without rewriting application logic.

# ingest.py
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

documents = SimpleDirectoryReader("./data").load_data()

index = VectorStoreIndex.from_documents(
    documents,
    embed_model=OpenAIEmbedding(model="text-embedding-3-small"),
)

index.storage_context.persist(persist_dir="./storage")

# query.py
from llama_index.core import load_index_from_storage, StorageContext
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.llms.openai import OpenAI

storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

retriever = VectorIndexRetriever(index=index, similarity_top_k=4)
query_engine = RetrieverQueryEngine.from_args(
    retriever=retriever,
    llm=OpenAI(model="gpt-4o-mini"),
)

response = query_engine.query("What's the refund policy?")
print(response)

Pitfalls:

• Abstraction overhead: simple things become verbose (see the query engine setup above)
• Chunking strategy defaults are often wrong for your domain — you’ll customize SentenceSplitter anyway
• Persistence format is opaque; migrating indices across versions breaks
• Async support is inconsistent across components

When to skip: You need custom retrieval logic (hybrid search, reranking, knowledge graphs) or your data doesn’t fit “documents” (structured DB rows, API responses, time-series).

Option 3: LangChain + LangGraph — best for complex chains and agents

LangChain’s LCEL (LangChain Expression Language) gives you composable pipelines with streaming, batch, and async support. LangGraph adds stateful, cyclic graphs for agents with persistence and human-in-the-loop.

# chain.py
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import JsonOutputParser
from langchain_core.runnables import RunnablePassthrough, RunnableLambda
from pydantic import BaseModel, Field

class Extraction(BaseModel):
    company: str = Field(description="Company name")
    amount: float = Field(description="Invoice total")
    date: str = Field(description="ISO date")

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
parser = JsonOutputParser(pydantic_object=Extraction)

prompt = ChatPromptTemplate.from_messages([
    ("system", "Extract invoice data. {format_instructions}"),
    ("human", "{text}"),
]).partial(format_instructions=parser.get_format_instructions())

chain = (
    {"text": RunnablePassthrough()} 
    | prompt 
    | llm 
    | parser
)

# Streaming works automatically
async for chunk in chain.astream(invoice_text):
    print(chunk, flush=True)

# agent.py
from langgraph.graph import StateGraph, END
from langgraph.checkpoint.sqlite import SqliteSaver
from typing import TypedDict, Annotated
import operator

class State(TypedDict):
    messages: Annotated[list, operator.add]
    iterations: int

def agent_node(state: State):
    # ... call LLM with tools ...
    return {"messages": [response], "iterations": state["iterations"] + 1}

def should_continue(state: State):
    return "tools" if state["iterations"] < 5 else END

workflow = StateGraph(State)
workflow.add_node("agent", agent_node)
workflow.add_node("tools", tool_node)
workflow.set_entry_point("agent")
workflow.add_conditional_edges("agent", should_continue)
workflow.add_edge("tools", "agent")

app = workflow.compile(checkpointer=SqliteSaver.from_conn_string("checkpoints.db"))

# Resumable, interruptible execution
config = {"configurable": {"thread_id": "user-123"}}
for event in app.stream({"messages": [("user", "Research quantum computing")]}, config):
    print(event)

Pitfalls:

• Verbosity tax: 50 lines for what raw API does in 10
• Version churn: LCEL, Runnable, LangGraph — three paradigms in two years
• Debugging opaque runnables is harder than reading your own code
• Checkpointing adds SQLite dependency; Postgres requires extra setup

When to skip: Simple request/response, single-tool chat, or when you can express the logic in 20 lines of Python/TypeScript.

Option 4: Raw HTTP + validation — best for everything else

Most side projects don’t need a framework. The OpenAI-compatible API surface is stable. Add Zod/Pydantic for schemas, httpx/fetch for requests, and you control everything.

# client.py
import httpx
from pydantic import BaseModel
from typing import Literal

class ChatMessage(BaseModel):
    role: Literal["system", "user", "assistant", "tool"]
    content: str | None = None
    tool_calls: list | None = None

class ChatRequest(BaseModel):
    model: str
    messages: list[ChatMessage]
    temperature: float = 0.7
    max_tokens: int = 1000
    tools: list | None = None
    tool_choice: Literal["auto", "none"] | dict = "auto"

class ChatResponse(BaseModel):
    choices: list[dict]
    usage: dict

async def chat_completion(request: ChatRequest) -> ChatResponse:
    async with httpx.AsyncClient(timeout=60.0) as client:
        resp = await client.post(
            "https://api.openai.com/v1/chat/completions",
            headers={"Authorization": f"Bearer {OPENAI_API_KEY}"},
            json=request.model_dump(exclude_none=True),
        )
        resp.raise_for_status()
        return ChatResponse(**resp.json())

# Streaming
async def stream_chat(request: ChatRequest):
    request.model_dump()["stream"] = True
    async with httpx.AsyncClient(timeout=60.0) as client:
        async with client.stream(
            "POST",
            "https://api.openai.com/v1/chat/completions",
            headers={"Authorization": f"Bearer {OPENAI_API_KEY}"},
            json=request.model_dump(exclude_none=True),
        ) as resp:
            async for line in resp.aiter_lines():
                if line.startswith("data: "):
                    data = line[6:]
                    if data == "[DONE]":
                        break
                    yield data

// types.ts
export const ToolCallSchema = z.object({
  id: z.string(),
  type: z.literal("function"),
  function: z.object({
    name: z.string(),
    arguments: z.string(),
  }),
});

export const MessageSchema = z.object({
  role: z.enum(["system", "user", "assistant", "tool"]),
  content: z.string().nullable(),
  tool_calls: z.array(ToolCallSchema).optional(),
});

export const ChatRequestSchema = z.object({
  model: z.string(),
  messages: z.array(MessageSchema),
  temperature: z.number().default(0.7),
  max_tokens: z.number().default(1000),
  tools: z.array(z.any()).optional(),
  tool_choice: z.union([z.literal("auto"), z.literal("none"), z.object({})]).default("auto"),
});

Why this wins for side projects:

• Zero abstraction leakage — you see every token sent and received
• Switch providers by changing the base URL (n4n.ai, Together, Fireworks, local vLLM all speak OpenAI format)
• No framework updates breaking your code
• Easy to add logging, caching, retries, fallbacks exactly how you want
• TypeScript/Python types are your contract

Pitfalls:

• You build conversation management, tool execution loops, streaming parsers
• No built-in prompt templates or few-shot management
• RAG and agent patterns are DIY

Common traps

Trap 1: “I’ll start simple and add a framework later”

Frameworks impose structure. Retrofitting LangChain into a raw API codebase means rewriting your orchestration layer. Pick the complexity ceiling upfront.

Trap 2: Over-indexing on “multi-provider support”

Every framework claims this. In practice, you pick one provider (or a gateway like n4n.ai that normalizes 240+ models behind one endpoint) and stay there. The switching cost isn’t the API — it’s prompt tuning, eval sets, and cost monitoring.

Trap 3: Using agents when you need deterministic pipelines

Agents are non-deterministic by design. If your side project processes invoices, generates reports, or extracts structure — use chains with validated outputs. Reserve agents for open-ended research or coding tasks.

Trap 4: Ignoring eval from day one

You cannot iterate on prompts without eval. Before adding a framework, write a 20-line script that runs your test cases and prints pass/fail. Frameworks add eval modules later; you need the discipline now.

# eval.py — do this first
import json
from client import chat_completion, ChatRequest

TEST_CASES = [
    {"input": "Invoice from Acme Corp for $1,234.56 on 2024-01-15", "expected": {"company": "Acme Corp", "amount": 1234.56, "date": "2024-01-15"}},
    # ... 20 more cases
]

async def run_eval():
    for i, case in enumerate(TEST_CASES):
        req = ChatRequest(model="gpt-4o-mini", messages=[{"role": "user", "content": case["input"]}])
        resp = await chat_completion(req)
        extracted = json.loads(resp.choices[0]["message"]["content"])
        passed = extracted == case["expected"]
        print(f"Test {i}: {'PASS' if passed else 'FAIL'} — {extracted}")

if __name__ == "__main__":
    import asyncio
    asyncio.run(run_eval())

My default stack for a new side project today

The gateway shortcut

If you want model flexibility without framework lock-in, put a gateway in front. One OpenAI-compatible endpoint, 240+ models, automatic fallback when a provider degrades, per-token metering, and it forwards provider cache-control hints so your Cache-Control headers actually work. You swap models by changing a header, not rewriting code.

Final rule

Choose the lowest abstraction that solves your problem.

• Need streaming chat in React? Vercel AI SDK.
• Need RAG over 500 PDFs? LlamaIndex.
• Need multi-step agent with persistence? LangGraph.
• Everything else? Raw HTTP + schemas.

You can always add a framework later. You can rarely remove one cleanly.