Every production chat system eventually hits the wall of a finite context window. To keep latency and cost sane, you must truncate conversation history context, but naive slicing drops instructions and user facts that the model still needs. This guide shows a deterministic pipeline to compress history while retaining the signals that matter.
Step 1: Measure and serialize your conversation state
You cannot truncate what you cannot measure. Serialize the chat into a list of dicts with role and content, then count tokens against the target model’s tokenizer. For OpenAI models, tiktoken is exact enough; for open-weight models, fall back to a character/4 heuristic but log the discrepancy.
import tiktoken
def count_tokens(messages, model="gpt-4o"):
enc = tiktoken.encoding_for_model(model)
# Approximates chat overhead; good enough for budget checks.
total = 0
for m in messages:
total += len(enc.encode(f"{m['role']}: {m['content']}"))
return total
messages = [
{"role": "system", "content": "You are a terse helper."},
{"role": "user", "content": "Book a flight to SF on Tuesday."},
{"role": "assistant", "content": "Sure, which airline?"},
]
print(count_tokens(messages)) # ~25 tokensStore the raw list. Never mutate the source of truth; produce a new truncated list each turn so you can replay and debug.
Step 2: Classify messages by retention priority
Not all turns are equal. Apply a fixed policy:
- System prompt: always keep.
- Last N exchanges (default N=6): keep verbatim—recent context drives the next reply.
- Older user/assistant pairs: candidate for summarization.
- Duplicate or purely acknowledgment turns (“ok”, “thanks”): drop silently.
def classify(messages, keep_recent=6):
labeled = []
# skip system at index 0
conversation = messages[1:]
for i, m in enumerate(conversation):
if len(conversation) - i <= keep_recent:
labeled.append((m, "high"))
elif m["content"].strip().lower() in {"ok", "thanks", "got it"}:
labeled.append((m, "drop"))
else:
labeled.append((m, "low"))
return labeled
labeled = classify(messages)This step makes the truncation decision auditable. A junior engineer can read the labels and see why a turn vanished.
Step 3: Summarize expired messages instead of dropping
Blind deletion loses commitments (“user said they are vegan”). Compress low priority turns with a cheap model call. Keep the summary under a token budget (e.g., 10% of window).
from openai import OpenAI
client = OpenAI() # swap base_url for any OpenAI-compatible gateway
def summarize(text, budget_tokens=120):
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": f"Compress chat to facts only. Max {budget_tokens} tokens."},
{"role": "user", "content": text}
]
)
return resp.choices[0].message.content
low_text = "\n".join(f"{m['role']}: {m['content']}" for m, pri in labeled if pri == "low")
summary = summarize(low_text) if low_text else ""The summary becomes a synthetic system note. It is not a conversation turn; it is context glue.
Step 4: Assemble the truncated context
Build the new message array: system prompt, optional summary block, then high-priority recent turns. This is where you actually truncate conversation history context without losing the spine of the dialogue.
def build_truncated(messages, summary, keep_recent=6):
system = messages[0]
out = [system]
if summary:
out.append({"role": "system", "content": f"Prior context summary: {summary}"})
# re-use classified high priority from step 2
labeled = classify(messages, keep_recent)
for m, pri in labeled:
if pri == "high":
out.append(m)
return out
trimmed = build_truncated(messages, summary)
assert count_tokens(trimmed) < count_tokens(messages)If you route through n4n.ai, the OpenAI-compatible endpoint forwards provider cache-control hints, so a stable summarized prefix stays cached even as recent turns rotate. That cuts repeat prefix costs on every request.
Step 5: Extract structured facts as a safety net
Summaries drift. For mission-critical state (user ID, booking params), run a JSON extractor on low turns and merge into a persistent memory dict that you inject as a system line.
import json
def extract_facts(text):
resp = client.chat.completions.create(
model="gpt-4o-mini",
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": "Extract key/value facts from chat. Output JSON."},
{"role": "user", "content": text}
]
)
return json.loads(resp.choices[0].message.content)
facts = extract_facts(low_text) if low_text else {}
if facts:
fact_line = "Known facts: " + json.dumps(facts)
trimmed.insert(1, {"role": "system", "content": fact_line})Now even if the summary paraphrases poorly, the raw slots survive.
Step 6: Wire into your inference call with fallback
Call the model with the trimmed list. Use a small wrapper that retries on rate limit and honors any routing header your gateway supports.
def chat(truncated_messages, model="gpt-4o"):
try:
resp = client.chat.completions.create(
model=model,
messages=truncated_messages,
temperature=0.2
)
return resp.choices[0].message.content
except Exception as e:
# delegate to gateway fallback or smaller model
if "rate" in str(e).lower():
return chat(truncated_messages, model="gpt-4o-mini")
raise
answer = chat(trimmed)Keep the original full messages in your DB. The truncated version is ephemeral per request.
Step 7: Verify success with replay tests
Truncation is a lossy transform; prove it loses the right things. Write a pytest that replays a 40-turn script, truncates, and asserts two properties:
- Token count is under your hard limit.
- A fact stated in turn 2 (e.g., “my seat is 12A”) appears in the final answer when queried.
def test_truncation_keeps_fact():
full = load_script("long_chat.json") # 40 turns, mentions seat 12A
summary = summarize(extract_low_text(full))
trimmed = build_truncated(full, summary)
assert count_tokens(trimmed, "gpt-4o") < 2000
ans = chat(trimmed + [{"role":"user","content":"What seat did I pick?"}])
assert "12A" in ansRun this in CI with a recorded mock of the summarizer to keep it fast. If the assertion fails, your summary prompt or budget is wrong—not the model.
Operational notes
- Log the
labeledpriorities for every truncated request for a week; you will find real conversations wherekeep_recent=6is too small. - Never summarize the system prompt. It is cheap and load-bearing.
- For multi-agent flows, truncate per agent, not globally.
Follow these steps and you will truncate conversation history context predictably, with tests that catch regressions before users do.