Token-efficient conversation summarization in LangChain

Token-efficient conversation summarization in LangChain is not a single API call — it’s a series of architectural decisions that compound across every request. Most teams start with ConversationBufferMemory, hit context limits, then bolt on ConversationSummaryMemory without understanding the tradeoffs. This guide walks through the ordered path from default behaviors to production-grade patterns, with code you can adapt directly.

Understand the memory hierarchy

LangChain provides three conversation memory classes that handle summarization differently. Each has distinct token profiles and latency characteristics.

ConversationBufferMemory stores every message verbatim. Token usage grows linearly with conversation length. Simple, predictable, and the default for a reason — but it fails hard once you exceed the model’s context window.

ConversationSummaryMemory maintains a running summary instead of raw messages. After each exchange, it calls an LLM to update the summary. Token usage stays roughly constant, but you pay an extra LLM call per turn and lose detail fidelity.

ConversationSummaryBufferMemory hybridizes both: it keeps recent messages verbatim (configurable by max_token_limit) and summarizes older ones. This is usually the right starting point for production workloads.

from langchain.memory import ConversationSummaryBufferMemory
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

memory = ConversationSummaryBufferMemory(
    llm=llm,
    max_token_limit=2000,          # keep ~2k tokens of recent history verbatim
    return_messages=True,          # return BaseMessage objects, not strings
    memory_key="chat_history",
)

The max_token_limit parameter is your primary tuning knob. Set it based on your model’s context window minus expected prompt overhead. For GPT-4o (128k context), 2000-4000 tokens leaves ample room for system prompts, RAG context, and response generation.

Choose the right summarization LLM

The summarization LLM does not need to be your primary reasoning model. Using GPT-4o to summarize conversations that GPT-4o-mini will later consume is wasteful. A smaller, faster model reduces latency and cost per turn.

from langchain_openai import ChatOpenAI

# Primary reasoning model
reasoning_llm = ChatOpenAI(model="gpt-4o", temperature=0.2)

# Dedicated summarization model — cheaper, faster
summary_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

memory = ConversationSummaryBufferMemory(
    llm=summary_llm,
    max_token_limit=3000,
    return_messages=True,
)

This separation also lets you tune temperature independently. Summarization benefits from temperature=0 (deterministic, factual). Reasoning tasks often need 0.2-0.7 for creativity.

Customize the summarization prompt

LangChain’s default summarization prompt is generic. For domain-specific conversations — support tickets, code review, medical triage — you lose critical entities and decision points. Override the prompt to preserve what matters.

from langchain.prompts import PromptTemplate

SUMMARY_PROMPT = PromptTemplate.from_template("""
You are maintaining a running summary of a technical support conversation.
Preserve: customer name, product version, error codes, troubleshooting steps tried,
proposed solutions, and any commitments made. Discard: greetings, pleasantries,
repeated information.

Current summary:
{summary}

New messages:
{new_lines}

Updated summary:
""")

memory = ConversationSummaryBufferMemory(
    llm=summary_llm,
    max_token_limit=3000,
    return_messages=True,
    prompt=SUMMARY_PROMPT,
)

The prompt receives two variables: summary (the current running summary, empty on first turn) and new_lines (formatted new messages since last summary). Your template must output only the updated summary — no preamble, no markdown.

Pitfall: prompt drift over long conversations

Even with a good prompt, summaries degrade over 50+ turns. The LLM gradually omits details it deems “redundant” but which become critical later (e.g., a specific error code mentioned once in turn 3). Two mitigations:

1. Periodic full-context refresh: Every N turns, rebuild the summary from the full verbatim history (if you have it) rather than incrementally updating.
2. Entity extraction sidecar: Run a lightweight NER pass on each message and append extracted entities to the summary as structured metadata.

# Example: periodic rebuild every 20 turns
class PeriodicRebuildMemory(ConversationSummaryBufferMemory):
    rebuild_interval = 20
    
    def save_context(self, inputs, outputs):
        super().save_context(inputs, outputs)
        self._turn_count = getattr(self, "_turn_count", 0) + 1
        if self._turn_count % self.rebuild_interval == 0:
            self._rebuild_summary_from_buffer()
    
    def _rebuild_summary_from_buffer(self):
        # Re-summarize from full buffer if available
        all_messages = self.chat_memory.messages
        if len(all_messages) > self.max_token_limit * 4:  # rough char estimate
            new_summary = self.llm.invoke(
                SUMMARY_PROMPT.format(summary="", new_lines=self._format_messages(all_messages))
            )
            self.moving_summary_buffer = new_summary.content

Implement token-aware truncation

max_token_limit uses a rough character heuristic (4 chars ≈ 1 token). For precise control, implement your own token counter using the model’s tokenizer.

import tiktoken
from langchain.memory import ConversationSummaryBufferMemory
from langchain_core.messages import BaseMessage

class TokenAccurateMemory(ConversationSummaryBufferMemory):
    def __init__(self, *args, model_name: str = "gpt-4o-mini", **kwargs):
        super().__init__(*args, **kwargs)
        self.encoding = tiktoken.encoding_for_model(model_name)
    
    def _get_token_count(self, messages: list[BaseMessage]) -> int:
        text = " ".join(m.content for m in messages)
        return len(self.encoding.encode(text))
    
    def _trim_buffer(self, messages: list[BaseMessage]) -> list[BaseMessage]:
        # Keep most recent messages that fit within token limit
        total = 0
        kept = []
        for msg in reversed(messages):
            msg_tokens = len(self.encoding.encode(msg.content))
            if total + msg_tokens > self.max_token_limit:
                break
            kept.insert(0, msg)
            total += msg_tokens
        return kept

This prevents the “off by 20%” surprise where your 3000-token limit actually consumes 3800 tokens because of Unicode, code blocks, or system prompt overhead.

Handle multi-turn tool use and structured outputs

Conversations with tool calls (function calling, ReAct agents) produce messages that don’t summarize cleanly. A tool call + tool result pair carries semantic weight that a generic summarizer drops.

from langchain_core.messages import ToolMessage, AIMessage

def format_tool_interaction(ai_msg: AIMessage, tool_msg: ToolMessage) -> str:
    """Format tool call/result for summarization prompt."""
    tool_name = ai_msg.tool_calls[0]["name"] if ai_msg.tool_calls else "unknown"
    args = ai_msg.tool_calls[0]["args"] if ai_msg.tool_calls else {}
    result = tool_msg.content[:500]  # truncate large results
    return f"[Tool: {tool_name}({args}) → {result}]"

Extend your summarization prompt to recognize this format:

Preserve: tool names, key arguments, success/failure, returned IDs or error codes.
Format tool interactions as: [Tool: name(args) → result]

Integrate with retrieval-augmented workflows

In RAG systems, conversation history competes with retrieved context for token budget. The summary should complement — not duplicate — what retrieval provides.

from langchain.chains import ConversationalRetrievalChain

qa_chain = ConversationalRetrievalChain.from_llm(
    llm=reasoning_llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    memory=memory,
    combine_docs_chain_kwargs={
        "prompt": PROMPT_WITH_HISTORY_AWARE_RETRIEVAL
    },
)

Design your summary prompt to focus on conversation state (what the user wants, decisions made, constraints expressed) while letting retrieval handle domain knowledge (documentation, specs, past tickets). This separation avoids the common failure mode where the summary re-states facts already in retrieved chunks.

Monitor token usage in production

You cannot optimize what you don’t measure. Log per-turn token consumption for both the summarization call and the main reasoning call.

import logging
from langchain.callbacks import get_openai_callback

logger = logging.getLogger("token_usage")

def log_token_usage(operation: str, callback):
    logger.info(
        f"{operation}: prompt_tokens={callback.prompt_tokens} "
        f"completion_tokens={callback.completion_tokens} "
        f"total_tokens={callback.total_tokens} "
        f"cost_usd={callback.total_cost:.6f}"
    )

# Wrap your chain invocation
with get_openai_callback() as cb:
    response = qa_chain.invoke({"question": user_input})
    log_token_usage("reasoning", cb)

with get_openai_callback() as cb:
    memory.save_context({"input": user_input}, {"output": response["answer"]})
    log_token_usage("summarization", cb)

Track these metrics over time:

• Summarization tokens per turn — should be stable; spikes indicate prompt issues
• Reasoning tokens per turn — should correlate with max_token_limit + retrieval size
• Summary quality proxy — user clarification requests (“I already told you…”) indicate summary loss

Common pitfalls and tradeoffs

Pitfall: summarizing too aggressively

Setting max_token_limit too low (e.g., 500 tokens) forces frequent summarization. Each summarization call adds 500-2000ms latency and accumulates drift. For most chat applications, 2000-4000 tokens balances context retention with context window safety.

Pitfall: losing code and structured data

Default summarizers destroy code blocks, JSON, stack traces, and table formatting. If your conversations include technical artifacts, either:

• Exclude them from summarization (keep verbatim in buffer)
• Use a prompt that explicitly preserves fenced code blocks and structured formats

TECHNICAL_SUMMARY_PROMPT = PromptTemplate.from_template("""
Preserve ALL code blocks, JSON, stack traces, and configuration snippets verbatim.
Summarize only natural language discussion.

Current summary:
{summary}

New messages:
{new_lines}

Updated summary:
""")

Pitfall: assuming summary memory works with all chain types

ConversationSummaryBufferMemory returns BaseMessage objects when return_messages=True. Some older chain types expect a string chat_history. Check the chain’s expected input format or use ConversationStringBufferMemory variants.

Tradeoff: latency vs. context fidelity

For latency-sensitive applications (real-time chat), consider async summarization: fire the summarization call non-blocking and accept slightly stale summaries.

import asyncio
from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=2)

async def async_save_context(memory, inputs, outputs):
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(executor, memory.save_context, inputs, outputs)

# In your request handler:
asyncio.create_task(async_save_context(memory, {"input": user_input}, {"output": response}))

This keeps the user-facing response fast while summarization happens in background.

Production hardening checklist

Before deploying summarization memory to production:

• Token budget documented: Calculate max_token_limit = model_context - (system_prompt + max_retrieval + max_response + safety_margin)
• Summarization model separated: Different model from reasoning, temperature=0
• Domain-specific prompt: Preserves entities, tool calls, structured data relevant to your use case
• Token-accurate truncation: Using tiktoken, not character heuristic
• Periodic rebuild strategy: Every 15-25 turns for long conversations
• Observability: Per-turn token logging, summary quality alerts
• Fallback behavior: If summarization fails, truncate buffer instead of dropping conversation
• Load tested: Verify summarization latency p99 under concurrent load

The n4n.ai gateway can help with the fallback layer — when your primary summarization provider is degraded, automatic fallback to a secondary model keeps the pipeline moving without code changes.

Summary

Start with ConversationSummaryBufferMemory at 2000-3000 tokens using a dedicated small model (GPT-4o-mini, Claude Haiku, or equivalent). Customize the prompt for your domain. Add token-accurate truncation with tiktoken. Implement periodic rebuilds for conversations exceeding 20 turns. Log everything. The difference between a prototype that works for 5 turns and a production system that handles 500 is almost entirely in these details.