Build a Memory System

The rest of this site is a comparison. This page is the route through it.

It exists because the material a builder needs was spread across four documents and two files that were never published, so the only way to get from my agent needs memory to here is what I should implement was to read a comparative report end to end and synthesise it yourself. Everything below already existed; what is new is the order, and the fact that you can now reach it.

Read what this page does not give you before you rely on it. Nothing here has been run against a working system by this project, and the page says so in the places where the omission matters.

%% caption: the five stages, gated on one question: if nothing has to survive the session, the answer is a prompt and a scratchpad rather than a memory system
flowchart TD
    Q["Does anything need to survive the session?"]
    Q -->|"no"| STOP["Build nothing.<br/>A prompt and a scratchpad are the answer"]
    Q -->|"yes"| P["1 · Pick the profile<br/>from the failure that hurts"]
    P --> B["2 · Write the brief<br/>and stop for approval"]
    B --> S["3 · Build in stages<br/>each one useful alone"]
    S --> T["4 · Verify by test id<br/>each says what it does not prove"]
    T -->|"a test fails"| S
    T --> C["5 · Close it out<br/>what is closed, what is open, what was deferred"]

1. Pick the profile from the failure, not the mechanism

Start from the failure your product cannot tolerate. Starting from the mechanism that sounds most rigorous is how a single-user note-taker acquires a tombstone it will never need, and the cost lands on whoever maintains it.

Two routers do this job, both on the pattern index:

• How to use the library — a list of symptoms, each pointing at the pattern that closes it. Wrong facts return after correction. Memory cannot be audited. Memories leak across projects. Retrieval repeats one document. Find yours and follow the link.
• Stacks, by what you are building — five product shapes, each naming the failure that actually hurts for that shape and the two or three patterns that close it. Single-user tool, multi-tenant, companion or roleplay, an autonomous agent that acts, and memory that must be correctable and defensible.

Patterns fail at their intersections rather than one at a time, which is why the unit is a stack. A rejected-value tombstone is decorative if three ungoverned write paths bypass the check. Hybrid retrieval is actively dangerous without scope as a key, because better recall means a wider blast radius when the boundary is missing.

The correctable stack is not the default. Scope, evidence, a governed gateway and a tombstone are the answer for memory that must be defensible under a correction. Most products are not that, and the pattern index says so in its own words.

And it is the route with the fewest worked examples. The pattern index states the thinning where it can be argued with: the tombstone is the rarest of the seven mechanisms, and no system in the atlas carries all four of these patterns. Building this stack means building it rather than copying it — which is the same fact as there is no reference implementation below, stated as a property of the advice instead of as a gap in the atlas.

And every one of those choices costs something. What you buy, and what it costs collects the six axes where the corpus shows these mechanisms trading against each other — recall against abstention, retention against correctability, scope against reuse — with the systems at each end and what the choice forced them to build. Read it if you are deciding between two profiles rather than following one.

The question before all of them

Does anything need to survive the session at all? If the answer is that the model needs the right things in this conversation, that is context assembly and not memory, and the smallest honest answer is a prompt and a scratchpad. The atlas keeps a scope boundary for exactly this case, because the two get conflated constantly and the second one costs an order of magnitude more to build.

2. Write the brief, then stop

Before any code: one page saying what is being built, which failure each part closes, and what is being deferred and why. The deferral list is the part worth more later than the code you write instead — it is the difference between a system that lacks a tombstone and a system whose author decided it did not need one.

The format is in .agents/protocol/build-brief.md: the brief, a closure report, and a lock file recording which atlas pages the decisions came from. It is written for an agent and reads fine as a checklist.

Stop here for a human decision. A memory layer is a schema plus a set of irreversible commitments about deletion and correction; those are cheap to change on one page and expensive to change afterwards.

3. Build in an order where each stage stands alone

Taken from §8 What I Would Build, which carries the table-level schema, the four-state status enum, the ten-step write path, the seven-step retrieval path and the context-assembly rules. The staging matters more than the schema: vector search and model-based extraction come last, deliberately, because a system that stores raw evidence and searches it lexically already works, and a system that starts with extraction has no floor to fall back to when extraction is wrong.

The order is argued from failure modes, not measured, and it has a case where it does not hold. It is derived from what breaks in this corpus — extraction that was wrong with nothing underneath it, scope retrofitted into a store that already had rows — and no reference implementation exists to adjudicate it, which is why that absence is listed below rather than buried. The case it does not fit is the one where retrieval quality is the product rather than a property of it. A tool whose whole value is finding the passage a lexical query misses has no shippable version at the end of Stage 1, and staging vectors last would mean withholding the product to defend against a failure it does not have yet. Read the ordering as a claim about where correctness debt accumulates, not as a release plan for every shape.

There is adjacent measurement, and it is worth reading with its distance stated. arXiv:2605.05242 — Beyond Semantic Similarity (Li et al., 3 May 2026) — has agents work a raw corpus with general-purpose tools rather than through a similarity interface, and reports gains over sparse, dense and reranking baselines on BRIGHT and BEIR. That is evidence that a lexical floor can beat a dense index at search over a fixed corpus. It is not evidence for this staging order, which is about a store an agent writes to and later has to correct — a write path the paper does not touch. It is cited here because the nearest measurement to a claim is worth knowing even when it does not reach it, and because a reader who finds it later should not have to guess whether this page had seen it. What still transfers in that case is the cheap-before-the-index half — scope as a schema key and raw evidence retained — because both cost far more to retrofit once an index and its embeddings exist than they do to put in first.

Stage 1 is not a tenant boundary yet, and shipping it as one is the mistake this table can most easily cause. A read filter keeps one scope's memories out of another scope's results; it says nothing about a caller that passes someone else's scope as an argument, or a consolidation pass that summarises across the line the read path enforces. Those are scope.caller_cannot_widen and scope.background_respects_boundary in the table below, they are separate tests because they fail separately, and the pattern index is blunt about the reason scope comes first: it "has to reach the schema, the indexes, the cache keys, and every background job", which is why retrofitting it into a store that already has data is the hardest migration on its list.

Each of the other stages has one too, and in every case it is the version of the stage that looks finished from the outside. These are the shapes the corpus returns most often, so each names a system that shipped it.

• Stage 2 — a status column no read path consults. Adding the enum is the visible half; the invisible half is that a status only exists if a write path can produce every value and a read path filters on it. memoir earns no trust-state mark for exactly this reason — the field exists, the reader respects it, and nothing sets it — and csm states the inverse as a rule after hitting it three times in one codebase: never let a read path filter on a status the write path cannot produce. Both fail silently, rendering an empty section rather than an error.
• Stage 3 — filtering at index time and calling it retrieval. Once vectors arrive, a memory can be excluded when it is embedded and still come back, because the vector was written before the status changed. The fix is to apply the same predicate where the candidate set is assembled, not only where rows are indexed; MindCache is the corpus's worked example of getting this right, applying its status filter at the embedding fetch as well as the embedding job. This is also the stage where the §1 warning bites: better recall without scope as a key is a wider blast radius, not a better system.
• Stage 4 — an audit that does not hear about memory, and a background job that crosses the line the read path holds. A mutation audit is judged by whether its actions have producers, not by whether the table exists; Agent Mesh ships a hash-chained log in which nine of twenty actions have none. The job half is scope.background_respects_boundary, and it is separate from the read-path test because consolidation summarising across a boundary is a leak the retrieval filter never sees.
• Stage 5 — a correction surface no user can reach. The tombstone and the review queue are the two mechanisms most often built and left unreachable. memoir-cli argues the merge semantics harder than anything else here and ships no writer for the absolute tombstone that any of its fourteen tools can call; alma-memory computes a review queue whose needs_review() has no callers; OmniIntelligence writes its kill-switch events with a required reason and actor, then reads them through a materialized view nothing refreshes. Ask what user action produces the state before building the machinery that consumes it.

What is safe to defer cuts across the stages rather than along them, so the table above is one of two axes and this is the other. From the pattern index: bi-temporal validity, hybrid retrieval fusion, decay and reinforcement, and source-diverse context are improvements to memory that already works, and none of them prevents a silent failure. Laid over the stages, that means:

Two consequences a reader following the table sequentially would otherwise hit. Stage 3 holds both the most deferrable item on this page and one of the four the tensions page calls close to free — fencing recall as data costs almost nothing and can ship in Stage 1. And Stage 5 mixes the deferrable with the whole point of the correctable profile; a product that put itself in that row defers bi-temporal validity, not the tombstone.

Operational rules that belong at the stage they apply to, from §10: keep local state inspectable while developing, use transactional storage for primary state and reserve flat JSON for exports, add background workers only once synchronous semantics are settled, version schemas from the start, and provide a repair/reindex path before you need one.

4. Verify by test id

Twenty acceptance tests are specified in .agents/protocol/tests.yaml, each with a stable id, a given/when/then specification against your own API, the atlas page it was derived from, and — the field that matters most — what a pass does not prove.

The profile you picked in §1 selects the subset. Each test names the pattern it came from, and the pattern index's stacks table names the patterns per product shape, so the routing is a join over both rather than a separate judgement. The profiles and stage fields in the catalogue carry it:

The stage column below says when each one should first pass, so a stage is finished when its tests are green rather than when its code is written.

Every entry also carries a positive_control, and it is not optional. Sixteen of these twenty assert only absence, refusal or a bound, which means a system whose recall path always returns the empty set satisfies them: every leak probe comes back empty, every rejected value stays absent, and k is trivially an upper bound. The control is the paired presence assertion that has to hold in the same fixture — the memory comes back under its own scope, the neighbouring value is still recalled, the prompt contains k records and not zero. Run the absence half alone and a green result is evidence of nothing, which is the exact charge this atlas levels at published benchmark numbers.

Two are worth singling out because they are the ones a bundled suite passes without noticing. correction.retraction_without_replacement covers the case where a user says "I misspoke" and supplies nothing in its place: there is no newer fact for recency to prefer, so a system with no negative memory has nothing to work with, and the contradiction test expects this row to fail almost everywhere. deletion.absent_from_shared_copies covers deleting an original after it was exported, synced or shared to a second scope — a different failure from derived artifacts inside one scope, and one a system routinely passes the first while failing the second.

For the full versions, benchmarks §6 carries the thirteen-step deletion sequence with a six-method adapter contract, and §7 the contradiction test with its five case shapes.

5. Close it out

A closure report naming what is closed, what is open, and what was deferred with the reason. Not a conformance statement: this project certifies nothing, and a list of which failure modes you closed is more useful to the next maintainer than a badge would be.

If you publish numbers, benchmarks §8 gives the eleven axes worth reporting — including the four nobody reports: forbidden-hit rate, the fraction of retrieved memories that survive into the actual prompt, write-to-readable lag, and index bytes as a multiple of source bytes. Two rules go with them. Assert the cutoff, because a @k figure that scores more than k results is not measuring what it says. And commit the results, not just the harness — a reproducible harness with no committed output reads as measured and is not.

What this page does not give you

Stated here rather than discovered later, because the gaps are as load-bearing as the content.

• Nothing is runnable. The twenty tests are language-independent specifications, not executable fixtures. There is no adapter you can install and no suite you can point at your system. Turning each given/when/then into a test in your own stack is work this project has not done and cannot size for you: most are a few assertions against your own API, and a handful need fixtures — a second scope, an export or sync path, a full index rebuild — which is where the cost actually sits.
• The atlas has run none of them against anything. Every specification here was derived from reading code, and no system in the corpus has been put through the deletion sequence by this project. A test's presence in the catalogue is an argument that it is worth running, not evidence that anything passed it.
• There is no reference implementation. §8 is a specification at the level of tables, states and path ordering. It does not carry API signatures, idempotency or concurrency rules, or index-rebuild semantics, and no worked end-to-end design exists for any of the five profiles.
• There are no operating numbers. The scorecard names the axes worth measuring — write cost, read cost, retrieval p50 and p99, write-to-readable lag, bytes per memory. It cannot tell you what to expect, because almost nothing in the corpus publishes them and this project has measured none of them itself. Inventing the figures would be worse than the gap.

If you are an agent

The workflow above is packaged as a skill: .agents/skills/use-the-atlas/. It runs in one of four modes — decide, design, review, build — and only the last one writes code, after its own approval step. It also carries the instruction this page opens with: do not read the reports. There are hundreds, and reading widely is how an agent ends up recommending the most interesting mechanism instead of the smallest sufficient one. Read a system report only when a pattern page cites it for the exact mechanism you are borrowing.

scripts/check_protocol.py validates the test catalogue against the pages it cites and runs in the site's own test suite, so a test whose source argument changes goes stale visibly rather than quietly.