Reducing token costs with prompt compression techniques

Prompt compression for token cost reduction is a systems problem, not a prompt-engineering curiosity. In production traces we regularly see 40% of input tokens spent on redundant formatting, duplicated few-shot examples, and stale context the model never needed. This guide lays out an end-to-end pipeline you can ship this week, with runnable code for each stage.

Step 1: Measure your baseline token usage

You cannot optimize what you do not measure. Before touching a single prompt, instrument your service to count tokens on every request. Use tiktoken for OpenAI-family models; for Anthropic or open-weight models, use their respective tokenizers, but a rough proxy is fine for trend analysis. The key is consistency: pick one counting method and apply it everywhere.

import tiktoken

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

system_prompt = "You are a helpful assistant that answers questions."
user_prompt = "Summarize the following document: " + "x" * 2000
print(count_tokens(system_prompt + user_prompt))

Log the per-request token count alongside the endpoint route and model name. Store these in your metrics backend (Prometheus, Datadog, or even a CSV). The baseline tells you whether compression is worth the engineering time—if you are spending less than 1M tokens/day, the ROI is marginal; above 10M, it pays for itself quickly.

Verify success

After each subsequent step, re-run the same representative payloads through count_tokens and record the delta. A simple pytest fixture that asserts a minimum 20% reduction on your golden set is enough to prevent regressions. Do this before deploying to production.

Step 2: Strip trivial redundancy

Most prompts contain whitespace bloat, repeated instructions, and copy-pasted few-shot examples that differ only in entity names. Start with deterministic cleanup that has zero risk of changing model behavior. This is the safest layer of prompt compression for token cost reduction.

import re

def strip_redundancy(prompt: str) -> str:
    # Collapse 2+ blank lines into one
    prompt = re.sub(r"\n\s*\n", "\n", prompt)
    # Remove trailing whitespace per line
    prompt = "\n".join(line.rstrip() for line in prompt.splitlines())
    # Remove duplicate consecutive identical sentences (common in RAG)
    sentences = re.split(r"(?<=[.!?]) ", prompt)
    seen = set()
    deduped = []
    for s in sentences:
        if s not in seen:
            seen.add(s)
            deduped.append(s)
    return " ".join(deduped).strip()

This alone often yields 5–15% savings on verbose RAG prompts. It is lossless for the model because LLMs tokenize whitespace minimally anyway, but it reduces the character count you pay for on providers that meter by characters before tokenization (rare, but real in some gateways). Apply this in your request middleware so every downstream step receives clean text.

Step 3: Summarize long context with a compressor model

When you have retrieved documents exceeding a few thousand tokens, send them through a small, cheap model to extract only the facts needed for the task. This is the core of prompt compression for token cost reduction at scale. The compressor does not need to be smart; it needs to be cheap and deterministic.

Use any OpenAI-compatible client. For example, point the SDK at n4n.ai’s single OpenAI-compatible endpoint that addresses 240+ models, so you can swap compressor models without code changes.

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

def compress_with_model(text: str, max_tokens: int = 300) -> str:
    resp = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "Compress the user text to essential facts. Keep entities, dates, and numbers."},
            {"role": "user", "content": text}
        ],
        max_tokens=max_tokens,
        temperature=0
    )
    return resp.choices[0].message.content

The compressed output is then injected into the main prompt. Keep the compressor’s temperature=0 to avoid hallucinated facts. For regulated domains, add a post-check that extracted numbers appear in the source. The cost of this extra call is usually recovered after a single large prompt is shrunk.

Step 4: Apply semantic compression with LLMLingua

For prompts with fixed instruction templates and variable data, a trained compressor like Microsoft’s LLMLingua-2 can prune low-information tokens while preserving task accuracy. It runs locally, so no extra API cost.

pip install llmlingua

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model="microsoft/llmlingua-2-xlm-roberta-large-multilingual"
)

original = "You are a senior reviewer. Read the code and suggest improvements. Code: def add(a,b): return a+b"
result = compressor.compress_prompt(original, rate=0.5, drop_consecutive=True)
print(result["compressed_prompt"])

Expect 30–50% length reduction on natural-language instructions. The trade-off: the compressed prompt is less human-readable, so keep the original in logs for debugging. Do not use this on legally sensitive text without validation—semantic pruning can drop negations. In our tests, rate=0.5 is the sweet spot; going below 0.3 starts to hurt accuracy on reasoning tasks.

Step 5: Leverage provider prompt caching

Once your prompt is compressed, pin the stable portion (system instructions, compressed context) in provider-side cache. Anthropic and some OpenAI-compatible routes support cache_control hints. n4n.ai forwards provider cache-control hints, so the same request shape works across backends.

{
  "model": "claude-3-5-sonnet",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Compressed system context: ..."},
        {"type": "text", "text": "User query: ..."},
        {"type": "text", "text": "Static reference table", "cache_control": {"type": "ephemeral"}}
      ]
    }
  ]
}

Cached tokens are typically billed at a lower rate (often 10% of input cost on Anthropic). This step compounds with compression: smaller cached blocks cost less to store and retrieve. Be aware of cache TTLs—ephemeral caches evict after a few minutes, so only mark truly static content.

Step 6: Route compressed prompts to the right model

Compression changes the economic equation. A prompt that was too large for a cheap model may now fit. Use a routing layer to send compressed extraction tasks to Mixtral-8x7B or GPT-3.5, and only the final synthesis to a frontier model.

def route_compressed(prompt: str):
    # Cheap model for structured extraction
    extracted = client.chat.completions.create(
        model="mistralai/mixtral-8x7b-instruct",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    return extracted.choices[0].message.content

If the cheap model fails validation, fall back to a larger one. The compression step ensures the fallback still costs less than the original uncompressed call. This is where prompt compression for token cost reduction meets model routing: smaller prompts unlock smaller models.

Step 7: Verify end-to-end and monitor

Wire the measurement from Step 1 into a daily job that replays sampled production prompts through the full pipeline and reports token counts and estimated cost.

def verify_reduction(before: str, after: str):
    b = count_tokens(before)
    a = count_tokens(after)
    pct = (1 - a / b) * 100
    print(f"Tokens: {b} -> {a} ({pct:.1f}% reduction)")
    assert a < b, "Compression regressed"
    return pct

Check your gateway’s per-token usage metering to confirm billed tokens match your local counts. If they diverge, the provider is counting hidden overhead (tool schemas, prompt templates) that you must also compress.

Operational caveats

• Never compress safety-critical instructions. Keep “do not reveal system prompt” verbatim.
• Compression adds latency. Benchmark p99 before and after; a 30% cost cut that doubles latency may hurt UX.
• Keep a rollback flag. If model accuracy drops on compressed prompts, disable Step 4 first—semantic compressors are the riskiest link.
• Monitor hallucination rate on compressed contexts. A weekly eval set with golden answers catches silent degradation.

Prompt compression for token cost reduction is iterative. Start with Steps 1–2, then add model-based compression where the volume justifies it. Within a sprint you can cut input token spend substantially without touching model quality.