How much less an AI reads to answer a question about your codebase: a benchmark

We measured the token cost of answering a question about a codebase from a structure-aware map instead of the source. About 5 to 9x fewer tokens per question across five real repos. Here's the method, the numbers, and what it does not prove.

An AI coding tool answers a question about your codebase by reading. The only real variable is how much it has to read. Ask a stateless agent "what breaks if I change verifyToken?" and it pulls in whole files, guesses at the architecture, and hopes the right context landed in the window. Every question pays that reading cost again.

So we measured it. Not a marketing number: a reproducible benchmark with the method, the fixtures, and the results on real repositories, including the ones that look bad for us.

The headline: ~5 to 9x fewer tokens per question

The realistic question is not "how big is the whole map versus the whole source." It is "to answer one question, how much does a tool read." So we run the real retrieval path (hybridSearch, the same search_codebase engine the product ships). It picks the top files a cold agent would read, then we compare two conditions on the same files: the map cards for those files versus their full source.

| Repo | Files | Fewer tokens per question | |---|---|---| | secondos | 313 | 4.7x | | gurulu | 1,989 | 5.6x | | tcg-terminal | 51 | 8.8x |

Same question, same files, measured. The map carries the interface (paths, signatures, routes, the call graph), not every line of every function, so answering means reading a compact card instead of the body.

The bounded-map property

Here is the part that does not show up in a single ratio. We ran the orientation cost (the tokens a tool must load to be oriented on a repo) across five real codebases from 51 to 2,302 files, under three conditions: the raw source, the hand-written docs a team keeps (README, docs, CLAUDE.md), and the SecondOS map.

| Repo (files) | Source | Hand-written docs | SecondOS map | vs source | vs docs | |---|---:|---:|---:|---:|---:| | secondos (313) | 440k | 72k | 2.6k | 166x | 27x | | primdb (663) | 1,430k | 213k | 2.6k | 542x | 81x | | tcg-terminal (51) | 168k | 3k | 0.7k | 252x | 4x | | gurulu (1,989) | 3,897k | 839k | 2.3k | 1,689x | 364x | | adem (2,302) | 2,652k | 339k | 2.2k | 1,208x | 154x |

The finding: the orientation map stays bounded at roughly 2 to 2.6k tokens regardless of repo size. The source and the docs both grow without bound as the codebase grows. The map does not, because structure compresses in a way that raw text does not. A 2,302-file app orients in about the same budget as a 313-file one.

What we do NOT claim

The honest posture matters more than the big number, so read this part.

• The ~193x figure is an upper bound, not the per-question win. Comparing the whole map to the whole source gives huge ratios (secondos around 166x, gurulu around 1,689x). That is real but it is not what answering a question costs. The per-question ~5 to 9x is the number that describes daily use, and it is the one we lead with.
• This measures context cost, not answer quality. Fewer tokens to load the right context is the claim. Whether the final answer is correct depends on the model doing the reasoning, which is not what this benchmark isolates.
• We report the graph numbers that embarrass us. Across seven real OSS repos we resolve the decidable internal calls at very different rates per language: Python 60 · PHP 57 · Rust 51 · Java 45 · TS 40 · Go 24 · Ruby 9 (percent resolved). That tail is instance-method dispatch, which we deliberately mark unknown rather than guess. A lower number here is a precision stance, not a coverage gap: in this domain a confident wrong caller costs more than an honest gap.
• It is our own benchmark, on purpose. The standard memory benchmarks (LOCOMO, DMR) measure conversational recall, remembering what a chat said. That is a different category. No one had a standard for "does a tool understand this codebase without re-reading it," so we defined one in the open.

For completeness, the quality dimensions on the synthetic fixtures: code search hits precision@1 of 100% (MRR 1.000) on 34 discriminating symbols, and memory recall hits precision@1 of 92% (MRR 0.960) on 48 distractor-rich memories, with 4 of 4 structural invariants passing (a fresh memory outranks a stale, expired, or conflicting one at equal similarity).

Reproduce it on your repo

The number you care about is yours, not ours.

secondos sync /path/to/your/repo
bun scripts/benchmark-context.ts /path/to/your/repo   # tokens per question
bun scripts/benchmark-3way.ts    /path/to/your/repo   # orientation cost

And the context that leaves your machine is a structural map, never your source bodies (K-S23). See the full methodology and the live scorecard.