# How we measure context cost — our benchmark, in the open

Everyone in this category cites a cherry-picked accuracy number. We publish the methodology and the real-repo results instead — including the unflattering ones.

Claims about AI memory are easy to make and hard to trust. So instead of a marketing number, we publish a reproducible [benchmark](/benchmark) — the method, the fixtures, and the results on real repos, including the ones that look low.

## What we measure

Four things that matter for *codebase* memory (not conversational memory):

1. **Context cost** — tokens to answer a question. For a set of real questions, hybrid search picks the top files; SecondOS feeds their map *cards* while a cold agent reads the same files' *full source*. Same files, map vs body. Result: **~5–9× fewer tokens per question**.
2. **Code search** — does the right symbol rank first? precision@1 100%, MRR 1.000 on a 34-symbol discriminating fixture.
3. **Memory recall** — does the right note come back, and does a fresh note outrank a stale one? precision@1 92%, MRR 0.960, plus hard structural invariants.
4. **Graph resolution** — the share of *decidable* internal calls we resolve across seven real OSS repos, reported honestly (Python 60% … Ruby 9%). A precision stance: we say `unknown` rather than guess.

## Why our own benchmark, not LOCOMO/DMR

The standard memory benchmarks measure *conversational* recall — remembering what a chat said. That's a different category. No one has a standard for "does your AI understand this codebase without re-reading it," so we defined one, in the open, on real repos.

## Reproduce it

```
secondos sync /path/to/your/repo
bun scripts/benchmark-context.ts /path/to/your/repo
```

The number you get is yours, not ours. That's the point — [see the full methodology](/benchmark).