Base, instruct, and chat: picking the right model variant

You land on a model card and see three variants: base, instruct, and chat. The instruct vs chat model distinction trips up more engineers than it should. This guide gives you an ordered decision path, concrete code patterns, and the failure modes we’ve seen in production.

Understanding the three variants

Base models are raw next-token predictors trained on internet-scale corpora. They complete text but don’t follow instructions reliably. Instruct models take a base model and fine-tune it on instruction-response pairs — they understand “summarize this” or “write a function that…” Chat models add another layer: conversation-format training with system/user/assistant roles, often with RLHF or DPO for helpfulness and safety alignment.

The training pipeline typically looks like:

base → supervised fine-tuning (instruct) → preference optimization (chat)

Each step narrows capabilities but improves usability for specific tasks. You lose some raw creativity and open-ended reasoning at each stage, but gain instruction following and conversational coherence.

When to use base models

Use base models when you need maximum flexibility and control, and you’re building your own prompting or fine-tuning pipeline. Common cases:

• Custom fine-tuning: You’re training a domain-specific model (legal, medical, code) and need the full parameter space unfrozen.
• Few-shot prompting with curated examples: You have high-quality demonstrations and want the model to pattern-match without instruction-following bias.
• Logprob analysis and calibration: Base models produce better-calibrated probabilities for classification or routing tasks.
• Research and ablation: You’re measuring the effect of each training stage.

Pitfall: Base models hallucinate more on instruction-like prompts. They’ll complete your prompt as if it were a document prefix, not answer it.

# Base model completion — note the lack of instruction following
prompt = "### Instruction:\nWrite a Python function to parse JSON\n### Response:"
# Output often continues as: "\n\n### Instruction:\nWrite a test case...\n### Response:"
# instead of actually writing the function

If you’re not fine-tuning or doing careful few-shot engineering, skip base models.

When to use instruct models

Instruct models are your workhorse for single-turn, task-oriented workloads. They follow instructions without needing conversation formatting. Use them for:

• Structured extraction: “Extract entities as JSON with these fields…”
• Code generation: “Write a FastAPI endpoint that validates JWTs…”
• Summarization, translation, classification: Any well-defined transformation.
• Agent tool calls: When you need reliable function-calling format adherence.

Instruct models handle system prompts well but don’t expect multi-turn memory. Each request is independent unless you manually manage history.

# Instruct model — clean single-turn instruction following
messages = [
    {"role": "system", "content": "You are a precise code generator. Output only valid Python."},
    {"role": "user", "content": "Write a retry decorator with exponential backoff and jitter."}
]
# Returns clean function, no conversational filler

Tradeoff: Instruct models can be verbose. They often explain what they’re doing before doing it. Add “Output only the result” or use a strict system prompt to suppress this.

When to use chat models

Chat models excel at multi-turn conversations, roleplay, and scenarios where conversational tone matters. Use them for:

• Customer-facing chatbots: They maintain persona, handle chit-chat, and refuse appropriately.
• Coding assistants with context: Multi-file edits, debugging sessions, architectural discussions.
• Creative writing and brainstorming: The RLHF layer helps with open-ended collaboration.
• Human-in-the-loop workflows: Where a human reviews and continues the conversation.

Chat models expect the messages array with alternating roles. They’re sensitive to system prompt design — a weak system prompt degrades output more than on instruct variants.

# Chat model — multi-turn with context
messages = [
    {"role": "system", "content": "You are a senior Python engineer. Be concise."},
    {"role": "user", "content": "How do I handle connection pooling in asyncpg?"},
    {"role": "assistant", "content": "Use `create_pool` with `min_size` and `max_size`..."},
    {"role": "user", "content": "Show me a context manager pattern for acquiring connections."}
]
# Maintains context, references previous answer naturally

Pitfall: Chat models over-refuse on edge cases. Safety tuning can trigger on legitimate technical content (e.g., “exploit” in a security context). Test your domain-specific prompts thoroughly.

Common pitfalls and tradeoffs

Mixing variants in the same pipeline creates inconsistency. If your classifier uses a base model for logprobs but your generator uses a chat model, you’ll see format mismatches. Pick one variant per pipeline stage.

Assuming chat > instruct > base for all tasks. Chat models lose some instruction-following precision from the conversational training. For strict JSON output or function calling, instruct often beats chat.

Ignoring tokenizer differences. Some model families use different tokenizers across variants (e.g., chat adds special tokens for roles). This breaks logprob comparison and can cause off-by-one errors in token counting.

# Dangerous: assuming token counts transfer across variants
base_tokens = base_tokenizer.encode(prompt)
chat_tokens = chat_tokenizer.apply_chat_template(messages, tokenize=True)
# Lengths differ — special tokens, different whitespace handling

Over-relying on system prompts with chat models. A 2000-token system prompt consumes context and degrades attention to the actual task. Keep system prompts under 500 tokens; move domain knowledge to RAG or few-shot examples.

Quantization sensitivity varies. Base models often tolerate 4-bit quantization better — they have more redundancy. Chat models, with their finer-tuned distributions, can degrade noticeably at low bit-widths. Test your specific quantization level per variant.

Decision checklist

Follow this order for each new task:

1. Is this a training or fine-tuning workload? → Base model.
2. Single-turn, well-defined task with structured output? → Instruct model.
3. Multi-turn, conversational, or human-facing? → Chat model.
4. Need calibrated probabilities or logprobs? → Base model (or instruct with temperature=0).
5. Function calling or tool use required? → Check which variant the provider recommends for tool use — some chat models have dedicated tool-calling fine-tunes.
6. Latency-critical with strict format? → Instruct model with constrained decoding (JSON schema, regex).

def select_variant(task_type: str, requires_conversation: bool, 
                   needs_logprobs: bool, needs_tools: bool) -> str:
    if needs_logprobs:
        return "base"
    if task_type in ("classification", "extraction", "code_gen", "translation"):
        return "instruct"
    if requires_conversation or task_type in ("chatbot", "coding_assistant", "creative"):
        return "chat"
    # Default fallback
    return "instruct"

Quick reference code patterns

Instruct: structured JSON extraction

from pydantic import BaseModel
import instructor
from openai import OpenAI

class Invoice(BaseModel):
    vendor: str
    total: float
    line_items: list[dict]

client = instructor.from_openai(OpenAI(base_url="https://api.n4n.ai/v1"))
invoice = client.chat.completions.create(
    model="meta-llama/llama-3.1-70b-instruct",
    response_model=Invoice,
    messages=[{"role": "user", "content": ocr_text}],
    temperature=0
)

Chat: streaming conversation with tool calls

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1")
tools = [{
    "type": "function",
    "function": {
        "name": "query_database",
        "parameters": {"type": "object", "properties": {"sql": {"type": "string"}}}
    }
}]

stream = client.chat.completions.create(
    model="meta-llama/llama-3.1-70b-chat",
    messages=messages,
    tools=tools,
    tool_choice="auto",
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.tool_calls:
        # Handle tool call streaming
        pass
    elif chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Base: logprob-based routing

import numpy as np
from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1")

def route_by_confidence(prompt: str, labels: list[str]) -> str:
    completion = client.completions.create(
        model="meta-llama/llama-3.1-70b-base",
        prompt=prompt,
        max_tokens=1,
        logprobs=5,
        temperature=0,
        echo=True
    )
    # Extract logprobs for each label token
    token_logprobs = completion.choices[0].logprobs.top_logprobs[-1]
    scores = {label: token_logprobs.get(label, -20) for label in labels}
    return max(scores, key=scores.get)

Final note

The variant choice compounds across your stack. A chat model for generation + base model for classification + instruct model for summarization means three different prompting conventions, three tokenizer quirks, and three failure modes to debug. Standardize on one variant per pipeline unless you have a measured reason not to. Most production workloads we see settle on instruct for backend tasks and chat for user-facing ones — base stays in the training cluster.