Common prompt engineering mistakes and how to fix them

Prompt engineering mistakes show up in production as flaky outputs, runaway token bills, and debugging sessions that last days. Most teams don’t ignore best practices — they just learn them the hard way, one incident at a time. Below are the seven patterns I see most often in real systems, each with a fix you can apply immediately.

1. Vague or underspecified instructions

“Summarize this document” is not a prompt. It’s a wish. The model has to guess length, audience, format, tone, and which details matter — and it will guess differently every time. In production, that variance becomes inconsistency you can’t test against.

Write constraints explicitly. Specify word count, output structure, audience expertise level, and what to exclude. If you need a JSON object, say so and provide the schema. If you need a three-sentence executive summary for a non-technical stakeholder, say that.

# Weak
"Summarize the quarterly report."

# Strong
"""Summarize the quarterly report in exactly 3 sentences for a non-technical CEO.
Focus on: revenue change vs prior quarter, top risk factor, and cash runway.
Output plain text only — no markdown, no bullet points."""

The strong version is testable. You can assert sentence count, check for forbidden formatting, and verify the three required topics appear. That turns prompt engineering into something resembling software engineering.

2. Ignoring context window limits and token economics

Teams stuff entire codebases, chat histories, or document corpora into prompts because “the model supports 128k tokens.” Then they wonder why latency spikes, costs explode, and the model starts hallucinating from the middle of the context.

Context is not free. Every token costs latency and money, and retrieval quality degrades well before the hard limit. The fix is retrieval-augmented generation with a budget. Estimate your token ceiling per request, then build a pipeline that ranks and truncates context to fit.

def build_context(query: str, docs: list[Document], budget: int = 8000) -> str:
    """Select and pack documents into a token budget, highest relevance first."""
    ranked = rerank(query, docs)  # your reranker of choice
    packed, total = [], 0
    for doc in ranked:
        tokens = count_tokens(doc.content)
        if total + tokens > budget:
            break
        packed.append(doc.content)
        total += tokens
    return "\n\n---\n\n".join(packed)

This pattern — rank, budget, pack — beats naive stuffing every time. It also makes your system auditable: you can log which documents made the cut and why.

3. Treating few-shot examples as optional decoration

Zero-shot works for simple classification. It fails for structured extraction, multi-step reasoning, or any task where the output format has implicit rules. Teams skip examples to save tokens, then spend hours post-processing malformed outputs.

Include 3–5 diverse examples that cover edge cases: empty fields, nested structures, ambiguous inputs. Put them in a dedicated examples block, not inline with instructions. This separates the what from the how and makes the prompt readable.

{
  "task": "Extract product mentions as JSON",
  "schema": {"type": "array", "items": {"type": "object", "properties": {"name": {"type": "string"}, "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]}}}},
  "examples": [
    {"input": "Love the new iPhone 15!", "output": [{"name": "iPhone 15", "sentiment": "positive"}]},
    {"input": "The keyboard is okay but the trackpad sucks.", "output": [{"name": "keyboard", "sentiment": "neutral"}, {"name": "trackpad", "sentiment": "negative"}]},
    {"input": "No hardware mentioned here.", "output": []}
  ],
  "input": "{{user_review}}"
}

Examples are the most token-efficient way to specify format. They also serve as regression tests — if the model drifts, your examples catch it.

4. Not structuring output for parseability

“Return JSON” produces JSON — sometimes. Other times you get markdown fences, explanatory text, or a JSON object wrapped in a string. Downstream parsers choke, and you add fragile regex cleanup code that breaks on the next model update.

Enforce structure at the prompt level and the code level. Use a schema validator (Pydantic, Zod, JSON Schema) and retry on failure. Better: use a grammar-constrained decoder or structured output API where available. If your provider supports response_format: { "type": "json_schema", "json_schema": {...} }, use it.

from pydantic import BaseModel, ValidationError
from openai import OpenAI

class Extraction(BaseModel):
    entities: list[dict]

def extract_with_retry(prompt: str, max_retries: int = 3) -> Extraction:
    client = OpenAI()
    for attempt in range(max_retries):
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            response_format={"type": "json_object"},
            temperature=0
        )
        try:
            return Extraction.model_validate_json(resp.choices[0].message.content)
        except ValidationError as e:
            if attempt == max_retries - 1:
                raise
            prompt += f"\n\nPrevious output failed validation: {e}. Output valid JSON only."
    raise RuntimeError("unreachable")

Zero-temperature plus schema validation plus retry eliminates the parseability class of bugs. The prompt fix is free; the code fix is a few lines.

5. Overloading a single prompt with multiple tasks

“Analyze sentiment, extract entities, classify urgency, and draft a reply” in one prompt produces mediocre results on all four. The model’s attention splits, errors compound, and you can’t optimize or test any subtask in isolation.

Decompose into a chain or graph of focused prompts. Each step does one thing well, passes structured output to the next, and can be evaluated, cached, or swapped independently. This is prompt chaining — not to be confused with chain-of-thought reasoning.

# Step 1: Classify
classification = classify_ticket(ticket_text)  # returns {"category": "billing", "urgency": "high"}

# Step 2: Extract (only runs for relevant categories)
if classification["category"] in ("billing", "refund"):
    entities = extract_billing_entities(ticket_text)

# Step 3: Draft (uses structured context from steps 1-2)
reply = draft_reply(
    ticket_text=ticket_text,
    classification=classification,
    entities=entities or {}
)

Chaining adds latency, but parallelizable steps (classification + extraction) can run concurrently. The payoff: each prompt is simpler, testable, and replaceable without rewriting the whole pipeline.

6. Skipping systematic evaluation

Teams ship prompts based on “vibes” — a handful of manual tests that look good. Then a model update shifts behavior, or a new edge case appears in production, and nobody notices until customers complain.

Build an eval harness from day one. Curate a test set of 50–200 representative inputs with expected outputs or rubric criteria. Run it on every prompt change and every model version bump. Track pass rate, latency, and cost per request.

# evals/test_sentiment.yaml
- input: "This product is amazing!"
  expect: {"sentiment": "positive", "confidence": ">=0.8"}
- input: "It works, I guess."
  expect: {"sentiment": "neutral", "confidence": ">=0.5"}
- input: "Total waste of money."
  expect: {"sentiment": "negative", "confidence": ">=0.8"}

# runner.py
import yaml
from evaluate import run_eval

cases = yaml.safe_load(open("evals/test_sentiment.yaml"))
results = run_eval(cases, prompt_version="v3.2", model="gpt-4o-mini")
print(f"Pass rate: {results.pass_rate:.1%}")
print(f"Avg latency: {results.avg_latency_ms:.0f}ms")
print(f"Cost per 1k: ${results.cost_per_1k:.4f}")

Store results alongside your code. A failing eval blocks merge. This turns prompt engineering from art into a disciplined engineering practice.

7. Hardcoding model-specific quirks instead of using portable patterns

“Add ‘think step by step’ for GPT-4” works until you switch to Claude, Llama, or a fine-tuned variant. Model-specific prompt hacks create technical debt that compounds every time you evaluate a new provider.

Use portable patterns: explicit reasoning blocks, structured scratchpads, and self-consistency checks that work across architectures. If you need chain-of-thought, request it in a structured field — not as a magic phrase.

{
  "reasoning": "Step-by-step analysis goes here. Be thorough.",
  "answer": "Final concise answer",
  "confidence": 0.92
}

This schema works on any model that follows JSON instructions. When you route requests across providers — something n4n.ai handles with a single endpoint and automatic fallback — portable prompts mean you don’t rewrite your prompt library every time you shift traffic.

Summary table

Fix these seven and your prompt layer becomes maintainable, observable, and portable — exactly what production LLM systems need.