Agent memory, RAG, and fine-tuning are different layers. Memory decides what survives across sessions. RAG retrieves evidence for a prompt. Fine-tuning changes model behavior. Confusing them creates systems that retrieve documents but forget decisions, or train behavior when the real need is inspectable, deletable state.
Last updated: June 20, 2026.
What problem does each layer solve?
Use memory for durable facts, RAG for grounded evidence, and fine-tuning for model behavior. A production agent may use all three, but each has a different failure mode.
| Layer | Primary job | Best question | Bad fit |
|---|---|---|---|
| Agent memory | Persist scoped facts and source-linked state | ”What should this agent remember later?” | Static reference lookup without lifecycle |
| RAG | Retrieve source material for one answer | ”Which evidence should answer this query?” | Long-term user facts with deletion needs |
| Fine-tuning | Shift model behavior or task style | ”How should the model behave by default?” | Frequently changing facts or private memory |
| Context engineering | Assemble the current working set | ”What should the model see right now?” | Durable storage by itself |
The OpenAI conversation state guide makes the application responsibility clear: conversation state must be managed outside the model call. LangChain memory concepts similarly distinguish short-term thread state from long-term memory. RAG and fine-tuning can support the system, but neither replaces the need for managed state.
When is RAG enough?
RAG is enough when the problem is source lookup. If the user asks, “What does this policy say?” or “Which ticket mentions the migration plan?”, retrieval plus citations may solve the job. You need chunking, indexing, ranking, and evidence presentation.
RAG is not enough when the agent must decide what to remember after an interaction, keep that memory under a tenant or user scope, mark it superseded later, or delete it on request. Those are memory lifecycle requirements.
The MCP architecture docs describe tools and resources as ways servers provide context to clients. A RAG server can expose a search tool. A memory server needs a broader surface: write, retrieve, inspect, update, and delete or disable memory.
When is memory enough?
Memory is enough when the remembered item is small, scoped, and source-linked. Examples include “the project uses pnpm”, “the customer asked for SSO next quarter”, or “this repo’s staging host is stored in the setup note.” You still need retrieval, but the central job is durable recall.
Memory is not a replacement for a document index. If the agent needs to answer from a large policy manual, the memory layer should retrieve source artifacts and citations rather than compress the entire manual into a remembered fact.
Good memory uses RAG techniques, but it adds a write path. Read Memory needs a write path for the architecture that decides what becomes durable.
When should you fine-tune instead?
Fine-tuning is useful when you want the model to follow a style, classify in a consistent format, or improve a repeated behavior across many examples. It is usually the wrong place to store facts that change, belong to one tenant, or must be deleted.
If a user asks you to remove a specific remembered fact, a fine-tuned model cannot expose the same delete handle as an application memory record. If a product price changes, a fine-tune is slower and less inspectable than a superseded source artifact. If a tenant leaves, model weights are not a convenient tenant boundary.
Use fine-tuning for behavior. Use memory for state. Use RAG for evidence.
How do these layers work together?
A durable agent often uses this sequence:
- The application stores source artifacts and memory records.
- The agent asks a memory server for relevant context.
- Retrieval finds source-linked candidates.
- The context engineering layer selects the evidence and instructions for the model call.
- The model answers using citations.
- The write path decides whether a new fact, decision, or correction should be remembered.
Anthropic’s context engineering guidance focuses on curating the information an agent receives. That is the right framing: context is the current working set. Memory is one durable source for that working set.
What are the common failure modes?
The most common failure is treating RAG as memory. A team indexes documents, builds a search tool, and then wonders why the agent forgets a preference stated in yesterday’s session. Search did not fail; there was no write path.
The second failure is treating fine-tuning as a knowledge base. The model may learn a pattern, but the team loses the ability to inspect source lineage and delete one fact.
The third failure is treating the context window as durable storage. It works until compaction, a new session, or a different client clears the working set. See Context Engineering vs Memory for the boundary.
What should you choose?
Choose by the operational question:
| You need | Use | Why |
|---|---|---|
| Recall a project decision next week | Memory | It persists beyond the current prompt |
| Answer from a policy document | RAG | It retrieves source evidence |
| Make outputs follow a house style | Fine-tuning | It shifts behavior across tasks |
| Fit the right facts into one model call | Context engineering | It assembles the working set |
| Delete a specific remembered fact | Memory with delete handles | It targets source and derived artifacts |
Answer Engine’s wedge is memory delivered through MCP, REST, and CLI, not a claim that one layer replaces the others. Start with Agent Memory: The Complete Guide, then use the failure-mode catalog when one layer is being blamed for another layer’s bug.