Where Does an AI Agent’s Memory Actually Live?
A taxonomy built on four questions: where information sits, in what form, how it reaches the model, and how long it survives.
Four Questions About Any Stored Item
“The agent should remember the customer.” I hear this sentence often in projects, and everyone in the room nods. Two weeks later it turns out that three different things were meant. The business unit was thinking of the dialogue history within a session. IT was thinking of a connection to the CRM. And someone had in mind retraining the model on company data.
These three options differ in cost, in latency, in whether an answer can be evidenced, and in whether the information can be removed again later. So it is worth taking the word “memory” apart before talking about architecture or vendors.
I use four questions for this, and they can be asked of any single item an agent works with:
- Where does it sit?
- In what form is it stored there?
- How does it reach the model?
- How long does it survive?
The first three are decided together. The location largely fixes the form, and the form fixes which access path is possible at all. The fourth behaves differently, more on that below. Figure 1 shows the first three together.
Four Locations
The most common misconception I meet in teaching is that a model learns what you tell it in conversation. Chat products invite that reading. The system appears to remember, so it must have stored something. It has learned nothing. What looks like memory is written into the prompt again on every call, either from the dialogue history or from a store the product keeps alongside it. The model itself is unchanged.
The model’s weights hold whatever ended up in the parameters during training. I call this parametric memory. It is always available and costs nothing extra at runtime. It cannot be addressed. Ask which document a statement came from and there is no answer, because the statement is in no document. It is distributed across millions of numbers.
The context window holds everything the model reads on a given call: the system instruction, the dialogue history, retrieved passages, tool outputs. This store is explicit, inspectable, and freely changeable between calls. It is also the only place the model sees directly.
The runtime state is what the runtime carries through a single run. Intermediate results, loop counters, cached tool responses. It belongs to the run and disappears with it.
External stores are document repositories, vector indexes such as pgvector or Qdrant, graph databases such as Neo4j, and the records in systems that are already running. They outlive sessions. They are also the only place where content can be deliberately created, changed, and deleted.
The list follows proximity to the model. Close to the model, access is cheap and control is weak. Further out, every request costs work, but you can intervene.
The Form Decides the Access Path
Whatever sits in a location sits there in a particular form, and that form fixes the access path. A vector index stores embeddings and is searched by similarity (Lewis et al. 2020). A knowledge graph stores entities and edges and is traversed. Structured records are looked up by identifier. Raw documents are the borderline case where either works, depending on what was built beforehand.
The access method follows from the storage decision. Dump customer history as free text into a vector index and the question of how many cases a customer had last quarter becomes difficult. That is an aggregation, not a similarity search. The cleanest relational structure, on the other hand, helps little when someone wants to know what a maintenance manual says, in substance, about a fault pattern.
In Figure 1 all four forms converge on a single one at the context window. What the model reads is text, even when it began as a graph edge or a table row. This conversion happens at the process boundary and is a source of error in its own right.
The Time Question Behaves Differently
One might assume that retention follows from location. Context window short, external stores long. It does not work that way.
A context window holds both a system instruction that survives an entire session and a retrieved passage discarded after one call. External storage ranges from a debug log expiring after a few days, through curated session summaries kept for weeks, to reviewed domain knowledge maintained for years. Same kind of store, three entirely different retention regimes, with correspondingly different review obligations.
Persistence is therefore a question of its own and not a property of the location. Every category of content needs an explicit decision about how long it stays and who checks that. This decision belongs in the architecture, not in an operating procedure added later.
Only One Location Has No Way Back
In projects the deletion question rarely comes up early. It comes up when someone from data protection reads the concept.
For external stores, deletion is solved. Remove the record, the chunks generated from it, and the corresponding vectors. This costs effort, and the design has to make those derivations findable in the first place. It is achievable and auditable.
For parametric memory there is no such path. What has been trained into the weights cannot be selectively taken out. Research on machine unlearning develops approximate procedures and reports substantial efficiency gains over the obvious alternative (Yao et al. 2024). Approximate removal, however, is a weaker promise than Article 17 GDPR requires. The dependable route is retraining from a clean state, which presupposes that the training data and the base model are still around.
Figure 1 makes this asymmetry deliberately visible. Every store has an arrow into the context window. Into the weights runs a dashed one-way arrow, and nothing runs back out.
Adapter methods are the exception. In low-rank adaptation the pretrained weights stay frozen. What is learned is a small, separate pair of matrices whose product is added to a layer’s output (Hu et al. 2021). That delta is an artifact of its own. It can be detached, and the knowledge it carries goes with it. Choosing between full fine-tuning and an adapter is therefore a governance decision as much as a cost question.
What This Means for Decisions
The taxonomy yields an ordering that I now apply almost every time.
Retrieval is the default. The reason is not technical superiority but the operational properties: content is individually changeable, answers can be traced to a source, and swapping models leaves the index untouched. The family of methods is well studied, from classical retrieval-augmented generation (Lewis et al. 2020) through token-level nearest-neighbour approaches (Khandelwal et al. 2020) to architectures that build retrieval into the model itself (Borgeaud et al. 2022).
Fine-tuning earns its place when the target is a pattern rather than a fact. Output format, house style, terminology, one narrowly defined recurring task. Wherever the content stays stable and nobody will later ask where something came from. Through the OpenAI or Azure OpenAI APIs this has become a manageable operation, which lowers the barrier and makes the governance question correspondingly more pressing.
Episodic memory is a special case of long-term storage. The distinction between semantic knowledge and the recollection of specific events comes from cognitive psychology (Tulving 1972). Agentic systems need a counterpart in order to refer back to earlier sessions. It is typically implemented as a stream of natural-language observations from which entries are selected by recency, importance, and relevance (Park et al. 2023). In this article’s terms that is not a further location but long-term storage with a particular kind of content.
For a concrete project I like to reduce all of this to three questions. Does the content change? Then it belongs in an external store, however attractive a fine-tuned model sounds. Does the answer have to be evidenced? Then there is no way around retrieval with provenance, because a fine-tuned model asserts without being able to show. Could someone demand that this content be deleted? Then it must not go into the weights, at most into a detachable adapter.
Conclusion
Memory in agentic systems is a series of decisions: where something sits, in what form, by which path it reaches the model, and how long it stays. The first three are connected. The fourth needs an answer of its own.
The most useful rule of thumb is this. Anything you might later need to get rid of belongs outside. The context window forgets by itself, an external store can be cleaned up. The weights do neither.
A fuller version of this taxonomy appears in the book on AI agents that Michael Hewing and I are publishing with Springer Nature.