How we measure

Everyone in this category cites a cherry-picked accuracy number. We'd rather show you the methodology and the real-repo results — including the ones that look low. Here is what we measure, how, and what we get.

Memory recall

The one that actually discriminates. Given a query, does the right memory come back first? We evaluate on a labeled set of query → correct-note pairs and report the standard retrieval metrics.

precision@1 is the share of queries whose top result is the correct note; MRR credits the rank of the right answer when it isn't first. Both come from the same fused lexical + semantic recall the product ships.

Code search

search_codebase against a fixture with known-correct targets: for each query, is the intended symbol the top hit?

100% on the fixture is a strong result and an honest one: it's a fixture, not the whole world. It tells you the retrieval is sound on the cases we curated — read it as “the machinery works,” not “it is perfect on your repo.”

Token cost — the map vs re-reading files

The whole pitch is answering questions about your code without re-reading it every session. So the number that matters is what a question actually costs. We measure it the fair way: for a set of real questions, hybrid search picks the top-6 relevant files; SecondOS feeds their map cards (path + signatures + doc comments, never bodies — K‑S23), while a cold agent reads the same files' full source. Same files, map vs body.

On this repo the per-question ratio is 4.7×; on a 1,600-file production app it's 5.6× (≈2,200 tokens of cards vs ≈12,500 of source, per question). We lead with the per-question ~5× because that is what you pay to answer something — not a best case. The 193× is the honest structural fact underneath it: the whole map is a summary, so it is ~193× smaller than the full repo source. Reproduce either on your own repo: bun scripts/benchmark-context.ts <repo> (per-question) or bun scripts/benchmark-tokens.ts <repo> (whole map).

Three conditions, five real repos

One repo is one data point. Here is the same orientation cost — the tokens an agent must load to be oriented on a codebase — across five real repositories, under three conditions: agent (no memory, re-reads the source), agent + memory (the hand-written docs a team keeps — unindexed, loaded wholesale, quick to rot), and agent + SecondOS (the map: structure only, K‑S23, auto-generated and query-scoped).

The map stays bounded — ~2–2.6k tokens no matter the repo size (51 files or 2,302), while both the source and the hand-written docs grow without bound. And unlike the docs, the map is query-scoped and stays fresh as the code moves; a hand-written CONTEXT.md is stale the moment someone forgets to update it. Per question, through the real retrieval path, that is 4.7× (secondos) · 5.6× (gurulu) · 8.8× (tcg-terminal) fewer tokens than reading the same files cold — tcg-terminal was pushed to the cloud and answered live through the product. Reproduce across your own repos: bun scripts/benchmark-3way.ts <repo…>.

Relationship graph on real repos

who_uses and impact_of answer “who calls this?” and “what breaks if I change it?” We ran them against seven real open-source repositories and measured the share of decidable internal calls the graph resolves.

Read these as a precision stance, not a coverage gap. who_uses / impact_of resolve the calls they can prove and never guess; the rest is dynamic and instance-method dispatch that no static tool resolves without running your code — and we'd rather say “unknown” than point you at a wrong caller. A lower number here doesn't mean we found less; it means we claimed only what we could stand behind. Every resolved edge carries a confidence rung — exact, resolved, heuristic — so you always know which is which.

That's why the spread is wide. Python, PHP, Rust and Java expose more statically decidable call sites; Go's interfaces and Ruby's heavily dynamic dispatch leave more calls genuinely undecidable without executing the program. We report the provable fraction rather than inflating it with guesses that would break your impact_of the moment you trusted them.

Prove it on your own repo

Better than trusting our numbers: reproduce them on your code. Everything above runs from two open scripts — the number you get is yours, not ours.

bunx @secondos/cli push .              # map your repo (source stays local)
bun scripts/benchmark-context.ts .     # tokens/question — the map vs re-reading files
bun scripts/benchmark-tokens.ts .      # the whole map vs the whole source

Why do we publish our own methodology instead of citing a standard? Because the standard memory benchmarks (LOCOMO, DMR) measure conversational recall — remembering what a chat said. That is a different category. No one has a standard for “does your AI understand this codebase without re-reading it” — so we define one, in the open, on real repos, and hand you the scripts to check us. Own the benchmark for the thing you actually do; don't borrow one built for a different job.

Next

See the MCP server for the tools these numbers describe, the product overview for how the graph and memory fit together, or security for the trust boundary.