CrewAI memory management is the difference between a crew that repeats itself and one that accumulates context across tasks. In a multi-agent system, each agent needs scoped access to conversation history, persistent facts, and shared entity state without blowing up token costs or leaking irrelevant context.
Step 1: Scaffold a crew with shared memory enabled
Start by installing CrewAI and creating a minimal crew. The simplest way to give every agent in a crew a shared memory space is to set memory=True on the Crew object. This instantiates short-term and long-term memory backends that all agents read from and write to.
pip install crewai==0.30.0 # or latestfrom crewai import Agent, Crew, Task
researcher = Agent(
role="Researcher",
goal="Find facts about the topic",
backstory="You search diligently",
allow_delegation=False,
)
writer = Agent(
role="Writer",
goal="Draft a report from findings",
backstory="You write concisely",
allow_delegation=False,
)
task1 = Task(description="Research X", agent=researcher)
task2 = Task(description="Write about X", agent=writer)
crew = Crew(
agents=[researcher, writer],
tasks=[task1, task2],
memory=True, # shared crew memory
verbose=True,
)When memory=True is set at the crew level, agents do not get isolated buffers. They share a short-term store (recent interactions) and a long-term store (persisted across runs if backed by durable storage). If you need per-agent isolation, set memory=True on individual Agent instances instead.
Step 2: Configure the embedder for memory retrieval
CrewAI memory uses embeddings to retrieve relevant past interactions. By default it calls OpenAI’s embedding API. If you self-host or use a gateway, pass an embedder config. This is also where you can route through an OpenAI-compatible inference gateway—n4n.ai exposes one endpoint for 240+ models and forwards provider cache-control hints, which complements CrewAI’s memory by avoiding re-embedding static context.
crew = Crew(
agents=[researcher, writer],
tasks=[task1, task2],
memory=True,
embedder={
"provider": "openai",
"config": {
"model": "text-embedding-3-small",
"api_base": "https://api.n4n.ai/v1", # OpenAI-compatible
"api_key": "YOUR_KEY",
},
},
)If you omit embedder, CrewAI falls back to the OPENAI_API_KEY env var. For on-prem, swap the provider to ollama or huggingface and adjust the config.
Step 3: Control short-term memory scope
Short-term memory in CrewAI is a RAG buffer over the crew’s recent messages. Left untuned, it retrieves the top-k most similar exchanges to stuff into the prompt. That can confuse agents if unrelated tasks share keywords.
You can constrain scope by setting memory_config on the crew or agent. The two levers that matter: retrieval limit and score threshold.
crew = Crew(
agents=[researcher, writer],
tasks=[task1, task2],
memory=True,
memory_config={
"short_term": {
"retriever": "recent", # or "semantic"
"top_k": 5,
"score_threshold": 0.7,
}
},
)retriever="recent" disables semantic search and just feeds the last N turns—cheap and predictable. semantic uses the embedder. Set top_k low (3–5) for narrow tasks; raise it for research crews that need broad recall.
Step 4: Persist long-term memory across process restarts
Default crew memory uses an in-memory SQLite store that dies with the process. For durable cross-session memory, point the long-term backend at a file or Postgres.
from crewai.memory.storage.ltm_sqlite_storage import LTMSQLiteStorage
storage = LTMSQLiteStorage(db_path="./crew_ltm.db")
crew = Crew(
agents=[researcher, writer],
tasks=[task1, task2],
memory=True,
memory_config={
"long_term": {
"storage": storage,
}
},
)Now facts the crew learns in run one are available in run two. Inspect the DB with sqlite3 crew_ltm.db ".tables" to confirm rows in long_term_memory.
If you run multiple crews on one machine, give each a distinct db_path unless you intentionally want them to share long-term memory.
Step 5: Enable entity memory for structured facts
Entity memory extracts named entities (people, organizations, dates) from conversations and stores them as key-value pairs. It is essential when later tasks need “what was the client’s name?” without re-reading transcripts.
Enable it per agent or crew-wide:
writer = Agent(
role="Writer",
goal="Draft a report from findings",
backstory="You write concisely",
memory=True,
entity_memory=True, # capture entities
)When entity_memory=True, CrewAI runs a lightweight extraction step after each agent action. The extracted entities are queryable via the agent’s entity_memory attribute:
print(writer.entity_memory.get("client_name"))Do not enable entity memory for agents that handle free-form creative text only—extraction adds latency and token spend for little gain.
Step 6: Verify memory is actually being used
A crew that claims to have memory but ignores it is worse than no memory. Verify with a two-task test where task two depends on a fact from task one, and the agents cannot see each other’s prompts directly.
task1 = Task(
description="State the secret code: 42. Remember it.",
agent=researcher,
)
task2 = Task(
description="What was the secret code from the earlier task? Output only the number.",
agent=writer,
)
crew = Crew(agents=[researcher, writer], tasks=[task1, task2], memory=True)
result = crew.kickoff()
assert "42" in result.raw, "Memory failed to propagate"If the assertion fails, check: (1) memory=True is set at the right level, (2) embedder API is reachable, (3) score_threshold isn’t filtering out the only relevant memory. Lower threshold to 0.3 temporarily during debugging.
For long-term persistence, run the crew once, then re-instantiate with the same db_path and a task that queries an old fact. If it answers correctly, storage works.
Step 7: Avoid common memory leaks
Memory accumulates. In long-running crews, the SQLite file grows and semantic retrieval slows. Three practices keep it sane:
- Rotate DB files per project phase:
db_path=f"crew_{phase}.db". - Periodically delete low-value rows:
DELETE FROM long_term_memory WHERE created_at < datetime('now','-30 days'); - Disable
entity_memoryon agents that don’t need it.
Also watch token billing. Each retrieved memory block is injected into the prompt. With top_k=10 and verbose backstories, you can triple input tokens. Set top_k based on measured need, not guesswork.
Verify success
You now have a crew where agents share short-term context, persist facts across restarts, and extract entities. Success criteria:
- Agent B answers a question whose answer was produced only by Agent A in a prior task.
- After restarting the Python process and reusing the SQLite path, the crew recalls a fact from the previous run.
writer.entity_memory.get("some_key")returns a value set byresearcher.
If those hold, your CrewAI memory management is correct. If not, trace the verbose=True logs—CrewAI prints which memory chunks it retrieves before each LLM call.
That’s the full pipeline. Memory in multi-agent systems is not magic; it’s a storage and retrieval problem with tight token budgets. Treat it like any other stateful service.