Evaluate RAG and agent-memory accuracy with a golden set over your own sources. Write questions with known answers, log what the retriever returned, inspect the source evidence, and score whether the final answer is supported. Do not borrow a vendor number from a different dataset.
Last updated: June 20, 2026.
Why not use a public benchmark number?
Public benchmark numbers can be useful for research, but buying decisions need your sources, your permissions, and your failure modes. A memory layer that performs well on a public task may still fail when a source is stale, a ticket is permission-bound, or the correct answer requires a superseded policy to be ignored.
OpenAI’s evals guide recommends defining the task, running test inputs, and analyzing results before iterating (OpenAI evals guide). Its evaluation best-practices guide also distinguishes generic model benchmarks from tests you design for your own LLM application (OpenAI evaluation best practices).
Agent memory needs that application-specific approach.
What should your golden set contain?
A useful golden set is small, concrete, and source-backed. Start with questions your team can judge by looking at the source material.
| Case type | Example | Why it matters |
|---|---|---|
| Direct source answer | ”What is the current API retry policy?” | Tests whether retrieval finds the right source. |
| Changed fact | ”What is the current launch date?” | Tests supersession or stale-source handling. |
| Permission boundary | ”Show customer renewal notes for Account A.” | Tests tenant and source scope. |
| Abstention | ”What was the board decision on an undocumented plan?” | Tests whether the agent refuses unsupported answers. |
| Multi-source synthesis | ”What changed between the ticket and the launch doc?” | Tests source joining and citation behavior. |
| Deletion/correction | ”Does this removed memory still appear?” | Tests lifecycle controls. |
Do not make the set too broad at first. A small set that engineers actually inspect is more useful than a large spreadsheet nobody trusts.
What should you log?
Log the retrieval path, not just the final answer. Without retrieval logs, you cannot tell whether a bad answer came from missing source data, weak search, bad ranking, scope filters, or generation.
Capture:
- Query text and user/workspace scope.
- Candidate source IDs and ranks.
- Returned summaries or snippets.
- Current or superseded state.
- Final answer and citations.
- Latency and tool/client surface.
- Human score and notes.
This is where agent-memory evaluation differs from generic chatbot evaluation. The evidence path matters as much as the answer text.
How should you score answers?
Keep the rubric simple:
| Score | Meaning |
|---|---|
| Supported | The answer is fully supported by retrieved source evidence. |
| Partially supported | The answer is directionally right but misses source detail or caveats. |
| Unsupported | The answer makes claims not present in returned evidence. |
| Stale | The answer uses older evidence when a newer source exists. |
| Scoped out | The answer should not be available to this user or tenant. |
| Abstained correctly | The agent said it lacked evidence when sources were insufficient. |
You can turn this into numeric metrics later. For launch evaluation, the categorical labels are easier to review and harder to overclaim.
Where does Answer Engine fit?
Answer Engine should frame retrieval logs and scorecards as the proof mechanism, not publish a premature number. Roadmap: the public Answer Engine number ships only after the gated scorecard and retrieval instrumentation are complete.
Until then, the honest claim is methodological: use source-aware memory, inspect returned evidence, and run a golden set over your own data. That is stronger than an unsupported marketing number.
How do you run the first evaluation?
- Pick one source set, such as product docs and tickets.
- Write a golden set with direct, stale, permission, abstention, and multi-source cases.
- Run each question through the memory layer.
- Save returned source IDs, ranks, and summaries.
- Score the answer with the rubric above.
- Fix the first failure class before expanding the set.
The first run will probably reveal source gaps or retrieval misses. That is the point. Evaluation is not a launch badge; it is the operating loop for making memory trustworthy.
What should you read next?
Read Why agents recommend stale or outdated information for retrieval failure modes, Agent Memory: The Complete Guide for memory architecture, and the comparison hub for product-fit columns to include in procurement.