The SecondOS Benchmark

How much it costs to give an AI your codebase

Our own open benchmark for structure-aware codebase memory — the category SecondOS is in. Measured on five real repositories, three conditions each. Only aggregate token counts leave the machine; never your source (K‑S23).

~5–9×

fewer tokens per question than a cold agent

~2.6k

bounded map — same at 51 or 2,302 files

100% / 0.96

code-search p@1 · memory-recall MRR

Three conditions, five real repos

The orientation cost — tokens an agent must load to understand a codebase — under three conditions: agent (no memory, re-reads the source), agent + memory (hand-written docs — unindexed, quick to rot), and agent + SecondOS (the map: structure only, auto-generated, query-scoped).

agent · sourceagent + memory · docsagent + SecondOS · map

secondos

AI memory (this repo) · 313 files

source 440kdocs 72kSecondOS 3k

primdb

infra platform · 663 files

source 1.4Mdocs 213kSecondOS 3k

tcg-terminal

collection app · 51 files

source 168kdocs 3kSecondOS 700

gurulu

product analytics · 1,989 files

source 3.9Mdocs 839kSecondOS 2k

adem

eng-ops platform · 2,302 files

source 2.7Mdocs 339kSecondOS 2k

Repository	agent (source)	+ memory (docs)	+ SecondOS	vs source	vs docs
`secondos` · 313 files	440k	72k	3k	169×	28×
`primdb` · 663 files	1.4M	213k	3k	550×	82×
`tcg-terminal` · 51 files	168k	3k	700	240×	4.1×
`gurulu` · 1,989 files	3.9M	839k	2k	1695×	365×
`adem` · 2,302 files	2.7M	339k	2k	1205×	154×

The map stays bounded — ~2–2.6k tokens no matter the repo size (51 files or 2,302), while both the source and the hand-written docs grow without bound: that is 193× smaller than the whole source on this repo, and 166–1,689× across the five. And unlike the docs, the map is query-scoped and stays fresh as the code moves.

Per question

Through the real retrieval path — the same engine behind search_codebase, same top‑K files, map cards vs full source.

4.7×

secondos · 313 files

5.6×

gurulu · 1,989 files

8.8×

tcg-terminal · 51 files · answered live via the product

Quality, not just token counts

100%

code search precision@1 · MRR 1.000 (34-symbol fixture, rerank on)

92%

memory recall precision@1 · MRR 0.960 (48 distractors, vector-only)

4 / 4

memory invariants: fresh > stale · expiry & conflict labeled

7 repos

graph resolution: Python 60 · PHP 57 · Rust 51 · Java 45 · TS 40 · Go 24 · Ruby 9 (% of decidable calls, never guessed)

Kept honest

Per-question is the headline

The bounded map vs whole source runs to 166–1,689×, but we label that the upper bound. What you pay to answer one question is the honest number.

Same lever, opposite effect

A reranker raises code-search precision but lowers memory recall −12pt — so it ships on for search, off for memory. We report both.

A precision stance

The graph tail is instance-method dispatch we mark unknown rather than guess. Fixtures resolve 100%, real repos less — we report the real number.

Why our own benchmark?

LOCOMO / DMR measure conversational recall — a different category. No standard exists for “does your AI understand this codebase,” so we define one in the open.

Reproduce it on your own repo

The number you get is yours, not ours.

secondos sync /path/to/your/repo
bun scripts/benchmark-3way.ts     /path/to/your/repo   # agent / +memory / +SecondOS
bun scripts/benchmark-context.ts  /path/to/your/repo   # tokens per question

Want the full methodology — fixtures, structural invariants, the unflattering numbers? Read how we measure →

Now run it on your own repo

One sync, and your AI reads ~5× less. Free on your machine, no credit card.

Install SecondOS See pricing