LangChain’s ConversationSummaryMemory compresses chat history into a running summary, letting you maintain context across thousands of tokens without stuffing the entire transcript into every prompt. When you route those summarization calls through n4n.ai, you get automatic fallback across 240+ models and per-token metering without changing your application code. This guide walks through wiring it up end to end.
Step 1: Install dependencies
You need LangChain core, the OpenAI integration (n4n.ai speaks the OpenAI API), and a summary-capable model. Use a recent LangChain version — the memory API stabilized in 0.1.x.
pip install "langchain>=0.1.0" "langchain-openai>=0.1.0" python-dotenvCreate a .env file with your n4n.ai credentials. The gateway uses an OpenAI-compatible base URL and a single API key that works across all routed providers.
# .env
N4N_API_KEY=sk-your-n4n-key
N4N_BASE_URL=https://api.n4n.ai/v1Step 2: Configure the chat model pointed at n4n.ai
Instantiate ChatOpenAI with the n4n.ai base URL. Pass model="auto" to let the gateway pick the best available model for summarization, or pin a specific model like gpt-4o-mini if you need deterministic pricing. The temperature=0 keeps summaries consistent.
# config.py
import os
from langchain_openai import ChatOpenAI
def get_summary_llm() -> ChatOpenAI:
return ChatOpenAI(
model="auto", # or "gpt-4o-mini", "claude-3-haiku", etc.
temperature=0,
api_key=os.getenv("N4N_API_KEY"),
base_url=os.getenv("N4N_BASE_URL"),
max_tokens=512, # summary output cap
)If you want explicit control over which provider handles the summarization request, add the extra_headers parameter with X-n4n-Route: provider=anthropic (or openai, google, etc.). The gateway honors client routing directives and forwards provider cache-control hints automatically.
Step 3: Build the conversation chain with summary memory
ConversationSummaryMemory requires an LLM to generate and update the summary. It stores the compressed history in memory.buffer and exposes load_memory_variables / save_context like any other memory class. Pair it with ConversationChain for a minimal working loop.
# chain.py
from langchain.chains import ConversationChain
from langchain.memory import ConversationSummaryMemory
from config import get_summary_llm
def build_chain() -> ConversationChain:
llm = get_summary_llm()
memory = ConversationSummaryMemory(
llm=llm,
memory_key="history",
return_messages=False, # string summary, not message list
)
return ConversationChain(
llm=llm,
memory=memory,
verbose=True, # prints prompt + summary each turn
)The verbose=True flag is invaluable during development — you see exactly what summary the model produces and what gets fed back into the next prompt.
Step 4: Run a multi-turn conversation
Drive the chain in a loop. Each predict() call appends the new exchange, triggers a summary update if the buffer exceeds the model’s context window, and returns the assistant’s reply.
# main.py
from chain import build_chain
def main():
chain = build_chain()
print("Chat started. Type 'exit' to quit.\n")
while True:
user_input = input("You: ").strip()
if user_input.lower() in {"exit", "quit"}:
break
response = chain.predict(input=user_input)
print(f"Assistant: {response}\n")
if __name__ == "__main__":
main()Run it:
python main.pyYou should see the prompt template grow with a history section that stays compact — typically 200-400 tokens regardless of how many turns you’ve had.
Step 5: Inspect the summary buffer directly
For debugging or persistence, pull the current summary out of memory without making another LLM call.
# inspect.py
from chain import build_chain
chain = build_chain()
# Simulate a few turns
chain.predict(input="My name is Alex and I work on distributed systems.")
chain.predict(input="I'm debugging a consensus bug in Raft.")
chain.predict(input="The leader election keeps flapping under network partitions.")
memory = chain.memory
print("Current summary:\n", memory.buffer)
print("\nMemory variables:", memory.load_memory_variables({}))Output shows a concise paragraph capturing the salient facts. This is what gets injected into every subsequent prompt.
Step 6: Persist the summary to disk
Long-running processes (bots, agents, CLI tools) should serialize the summary so a restart doesn’t lose context. The simplest approach: pickle the memory object or write the buffer string to a file.
# persist.py
import json
from chain import build_chain
SUMMARY_PATH = "conversation_summary.json"
def load_summary() -> str | None:
try:
with open(SUMMARY_PATH) as f:
return json.load(f).get("summary")
except FileNotFoundError:
return None
def save_summary(summary: str):
with open(SUMMARY_PATH, "w") as f:
json.dump({"summary": summary}, f)
def main():
llm = build_chain().llm
prior = load_summary()
memory = ConversationSummaryMemory(
llm=llm,
memory_key="history",
return_messages=False,
)
if prior:
memory.buffer = prior # seed with previous summary
chain = ConversationChain(llm=llm, memory=memory, verbose=True)
# ... run your conversation loop ...
# On each turn or on shutdown:
save_summary(memory.buffer)For production, swap the JSON file for Redis, Postgres, or your preferred KV store. The key point: you only need to persist memory.buffer, not the full message list.
Step 7: Handle token limits and model switching
Summary memory works because the summary stays small. But the prompt sent to the model includes the system prompt, the summary, and the new user input. If you chain many tools or use a verbose system prompt, you can still hit the context ceiling.
Two practical guards:
- Set
max_token_limiton the memory — LangChain will trigger summarization earlier. - Monitor the prompt length — wrap the chain to log token counts.
# guarded_chain.py
from langchain.memory import ConversationSummaryMemory
from langchain.chains import ConversationChain
from config import get_summary_llm
import tiktoken
ENCODER = tiktoken.encoding_for_model("gpt-4o") # close enough for token estimates
class TokenAwareConversationChain(ConversationChain):
def predict(self, input: str) -> str:
# Build the prompt the same way the parent does
prompt = self.prep_prompts([{"input": input}])[0]
token_count = len(ENCODER.encode(prompt.to_string()))
if token_count > 120_000: # leave headroom for response
raise RuntimeError(f"Prompt too large: {token_count} tokens")
print(f"[debug] Prompt tokens: {token_count}")
return super().predict(input=input)
def build_guarded_chain() -> TokenAwareConversationChain:
llm = get_summary_llm()
memory = ConversationSummaryMemory(
llm=llm,
memory_key="history",
return_messages=False,
max_token_limit=8000, # summarize when buffer > 8k tokens
)
return TokenAwareConversationChain(llm=llm, memory=memory, verbose=True)The max_token_limit parameter is your primary lever. Tune it based on your model’s context window and how much space you need for the current turn plus response.
Step 8: Verify success with a stress test
Write a script that hammers the chain with 50+ turns and asserts the summary stays bounded.
# test_stress.py
from guarded_chain import build_guarded_chain
def test_long_conversation():
chain = build_guarded_chain()
for i in range(60):
user_msg = f"Turn {i}: The user asks about topic {i % 5} with detail level {i}."
response = chain.predict(input=user_msg)
assert isinstance(response, str) and len(response) > 0
summary = chain.memory.buffer
token_count = len(ENCODER.encode(summary))
print(f"Final summary tokens: {token_count}")
assert token_count < 1000, "Summary grew too large"
print("Stress test passed.")
if __name__ == "__main__":
test_long_conversation()Run it:
python test_stress.pyYou should see prompt token counts plateau while the turn count climbs. The final summary stays under ~1k tokens regardless of conversation length.
Step 9: Swap the summarization model without code changes
Because n4n.ai routes across providers, you can change the summarization model by updating the model parameter in get_summary_llm() or by sending a routing header. No application logic changes required.
# Switch to a cheaper model for summarization only
def get_cheap_summary_llm() -> ChatOpenAI:
return ChatOpenAI(
model="gpt-4o-mini", # or "claude-3-haiku", "gemini-1.5-flash"
temperature=0,
api_key=os.getenv("N4N_API_KEY"),
base_url=os.getenv("N4N_BASE_URL"),
max_tokens=512,
)If the pinned model hits rate limits or degrades, the gateway automatically falls back to another healthy provider in the same tier. Your application sees a successful response either way.
Step 10: Meter usage per conversation
n4n.ai returns standard OpenAI usage fields (prompt_tokens, completion_tokens, total_tokens) in every response. Hook the callback to aggregate per-session or per-user.
# metered_chain.py
from langchain.callbacks.base import BaseCallbackHandler
from typing import Any, Dict
from guarded_chain import build_guarded_chain
class UsageTracker(BaseCallbackHandler):
def __init__(self):
self.prompt_tokens = 0
self.completion_tokens = 0
def on_llm_end(self, response: Any, **kwargs: Any) -> None:
usage = getattr(response, "llm_output", {}).get("token_usage", {})
self.prompt_tokens += usage.get("prompt_tokens", 0)
self.completion_tokens += usage.get("completion_tokens", 0)
def report(self) -> Dict[str, int]:
return {
"prompt_tokens": self.prompt_tokens,
"completion_tokens": self.completion_tokens,
"total_tokens": self.prompt_tokens + self.completion_tokens,
}
def main():
tracker = UsageTracker()
chain = build_guarded_chain()
chain.callbacks = [tracker]
# ... run conversation ...
print("Usage:", tracker.report())This gives you accurate per-conversation accounting for billing, quotas, or cost dashboards without parsing logs.
Common pitfalls
Summarization latency — Each summary update is an extra LLM call. For user-facing chat, run the summarization asynchronously or accept the ~500-1500ms overhead per turn. Batch updates (summarize every N turns) reduce calls but let the buffer grow larger.
Loss of nuance — Summaries discard exact phrasing, code snippets, and specific numbers. If your application needs verbatim recall (e.g., “what was the exact error message three turns ago?”), keep a sliding window of recent raw messages in addition to the summary. ConversationBufferWindowMemory(k=5) layered on top works well.
Prompt template drift — The default ConversationChain prompt includes The following is a friendly conversation... which wastes tokens. Override prompt with a minimal template:
from langchain.prompts import PromptTemplate
MINIMAL_PROMPT = PromptTemplate(
input_variables=["history", "input"],
template="Summary of conversation so far:\n{history}\n\nHuman: {input}\nAssistant:",
)
chain = ConversationChain(llm=llm, memory=memory, prompt=MINIMAL_PROMPT, verbose=True)Model mismatch — The summarization LLM and the conversation LLM can be different models. Use a cheap, fast model for summarization (gpt-4o-mini, claude-3-haiku) and a stronger model for the actual replies. Configure two ChatOpenAI instances pointed at the same n4n.ai endpoint with different model values.
What to do next
- Add a background job that periodically re-summarizes the full buffer with a larger context model for higher fidelity.
- Implement a hybrid memory: summary for long-term + buffer window for recent exact context.
- Wire the usage tracker into your observability stack (Datadog, Prometheus, OpenTelemetry).
- Test failure injection: kill the primary provider and verify n4n.ai falls back without your code noticing.
The combination of LangChain’s summary memory and n4n.ai’s multi-provider routing gives you a conversation layer that scales horizontally across models and vertically across token counts — without you managing the plumbing.