How we measure context cost — our benchmark, in the open

Everyone in this category cites a cherry-picked accuracy number. We publish the methodology and the real-repo results instead — including the unflattering ones.

Claims about AI memory are easy to make and hard to trust. So instead of a marketing number, we publish a reproducible benchmark — the method, the fixtures, and the results on real repos, including the ones that look low.

What we measure

Four things that matter for codebase memory (not conversational memory):

1. Context cost — tokens to answer a question. For a set of real questions, hybrid search picks the top files; SecondOS feeds their map cards while a cold agent reads the same files' full source. Same files, map vs body. Result: ~5–9× fewer tokens per question.
2. Code search — does the right symbol rank first? precision@1 100%, MRR 1.000 on a 34-symbol discriminating fixture.
3. Memory recall — does the right note come back, and does a fresh note outrank a stale one? precision@1 92%, MRR 0.960, plus hard structural invariants.
4. Graph resolution — the share of decidable internal calls we resolve across seven real OSS repos, reported honestly (Python 60% … Ruby 9%). A precision stance: we say unknown rather than guess.

Why our own benchmark, not LOCOMO/DMR

The standard memory benchmarks measure conversational recall — remembering what a chat said. That's a different category. No one has a standard for "does your AI understand this codebase without re-reading it," so we defined one, in the open, on real repos.

Reproduce it

secondos sync /path/to/your/repo
bun scripts/benchmark-context.ts /path/to/your/repo

The number you get is yours, not ours. That's the point — see the full methodology.