The hard part of an agent memory layer is deciding what to write

Every agent memory tool is built around retrieval. The bottleneck sits upstream: something has to decide what deserves to be remembered, and the human mostly does not. We counted on our own repo, and 4 of 40 decisions had been written down.

Every agent memory tool I have looked at closely is shaped like a retrieval problem. There is a store, an embedding, a recall call, and a benchmark about how often the right chunk comes back. That work is real. It is also the easier half. A retrieval system can only return what something already decided to write down, and the deciding is where memory actually dies.

We counted, on ourselves

This week I pointed our own mapper at our own repository and asked it one narrow question: for every decision it could identify, where was that decision written down? A document in the repo, a PR body, a commit message, or nowhere at all.

22 modules. 40 identifiable decisions.

• 4 in a real document in the repo
• 3 explained in a PR body
• 33 living only in a commit message

Now look at the conditions. The repo is three months old. It has one primary author. That author is building a product about this exact problem, with the capture pipeline running the whole time. That is close to the best case anybody gets, and the ratio is still 4 out of 40.

So when a tool tells you it gives your agent memory, the first question is not how good the recall is. It is what ever got written, and by whom.

The human is not going to write it

Not out of laziness. Three things stack against it.

You decide things mid-problem. The decision is a two second event inside an hour of work you are trying to finish, and writing it down is a separate act with its own overhead.

The payoff is displaced. A recorded decision is worth almost nothing to you today and a lot to whoever hits that code in six weeks, including future you, who will not remember this conversation.

And the worst one: at the moment you make a decision it feels too obvious to be worth recording. It only becomes valuable after you have forgotten it, which is exactly when you are no longer in a position to write it down.

A process that depends on someone overcoming all three, every time, is not a memory system. It is a hope.

"Record everything" fails in the other direction

The obvious fix is a faucet. Save every message, summarize every session, embed the lot. What you get is wallpaper: a growing pile of true-ish statements that costs tokens on every single request and buries the three things that actually mattered.

A study out of ETH Zürich found that hand-maintained context files can lower an agent's success rate while raising cost by more than 20%. Volume is not memory. Volume is a tax wearing a memory-shaped label.

So the write path needs judgment. Which is an unglamorous word for the part nobody puts in a demo.

What a write path with judgment actually does

Four stages, in the order they have to happen.

Read from where decisions get made. Not from a form somebody fills in. Decisions happen inside your AI sessions, in PR discussions, in commit messages, in the shape of the diff itself. Ours reads your local Claude Code, Codex and Cursor sessions when you opt in, plus git history, schema changes, and GitHub issues and PRs. If a memory layer only reads the chat, it will miss most of what you decided.

Cover the whole session, not its tail. This is the detail I underestimated. If you slice the last N tokens of a long working session, you capture almost nothing: in one 859-turn session of ours, an 11K token tail was 13 turns, about 2.3% of the conversation. Worse, a decision made at turn 200 and reversed at turn 700 becomes a lie if you only read one of them. So we fold instead of slicing: split the entire session into overlapping windows, carry a running state through them, and let each window revise or supersede what the earlier ones concluded. The final window is the answer, and it knows the conversation changed its mind partway through.

Reject before you store. Candidates pass a speculation filter, then dedup against what is already known, then a critic that returns a binary verdict: keep or reject. Deliberately no confidence score, because a number invites you to tune a threshold instead of making a decision, and a memory you are 61% sure about is not a memory. Most candidates die at this stage. That is the stage working.

Bind every survivor to real structure. A note that says "retries are capped at 3" attached to nothing is trivia. Attached to the symbol it constrains, it can surface when someone touches that symbol, and it can be flagged when that symbol changes. When we cannot tell what a decision belongs to, we bind it coarser, to the repo rather than to a guess, because a wrong anchor is more expensive than a vague one.

Then there is the consent question, which I had backwards

I assumed the safe design was to ask. The agent proposes, a human reviews, nothing is recorded without approval. It sounds obviously correct, and we built the review inbox first for exactly that reason.

Then 4 out of 40 happened. The human already does not do this. A review queue you never open is not consent. It is a slower way of remembering nothing.

So the real question is not whether the AI asks permission before recording. It is who is responsible for noticing when a recorded thing stops being true. If nothing is, then an AI writing its own memory is dangerous, and so is a human writing it, which is exactly what a stale CLAUDE.md already is. If something is, the cost of a wrong note collapses, and the thing to be afraid of is the missing note.

That is why the design is a net rather than a gate. Memories are anchored, they go stale on their own when the anchored code changes, contradictions supersede each other, and anything can be pulled in one click. Letting it record without review is still opt in and still off by default. I just no longer think leaving it off is the cautious choice. It is the choice where you end up with 4 out of 40.

Four questions for any memory layer, including this one

1. Where does it read from? If the answer is only the chat, it never sees most of what you decided.
2. What does it throw away? A store with no rejection stage is a context tax.
3. What is each memory attached to? If the answer is "a vector", nothing can tell you when it went false.
4. What happens when a memory becomes wrong? If the answer is nothing, you are not building an asset, you are accruing a liability.

What this does not do

It is judged by a model, so it both misses real decisions and lets some noise through. It is worth less on day one than on day thirty, because an empty memory is worth nothing: early on we asked our own system a "why" that had never been recorded anywhere and got exactly what it deserved, which was nothing. And it costs LLM calls on our side rather than yours, which is a choice we can defend because we measured it and the upkeep runs under a dollar a month per project.

The proof I actually trust is not a retrieval benchmark. It is that our own project went from a handful of memories to more than a hundred active ones in about two weeks, and that I typed almost none of them.

Get your own number

You can run the count on your own repo without an account or an install. Copy the prompt at secondos.app/scan/try, paste it into whatever AI coding tool you already have open, and paste the JSON it returns back into the page. Your agent reads your source locally and emits structure only, so no code leaves your machine.

Then compare: how many of your decisions live anywhere other than a commit message? If your ratio is better than 4 out of 40, I would like to know what you are doing differently.