Context-Dependence Benchmark
Context
How much does an agent task depend on carried context — and how well does your memory/context backend supply it? Fingerprint any backend across sections by knowledge type, with published negative controls, a /100 composite, and a first-class context-token bill. Backend-agnostic: bring your own backend and your own model.
Leaderboard
The /100 composite
Summed over the scored sections (S2–S6, 20 probes each), the S1 leak-gate excluded. The point is the fingerprint below, not one number — but this is the at-a-glance summary per backend, with the token bill it cost to supply the context.
| Backend | Model | Score | Context tokens | $ cost | correct / 1k-ctx | Provenance | Drift | Attribution |
|---|---|---|---|---|---|---|---|---|
oracle | synthetic | 100/100 | — | — | — | 20/20 | 20/20 | self_reported |
compaction | claude-sonnet-5 | 97/100 | 920 | $4.3070 | 105.435 | 0/20 | 20/20 | self_reported |
vendor-native | claude-sonnet-5 | 97/100 | 8428 | $4.4706 | 11.509 | 0/20 | 20/20 | self_reported |
rag-bm25 | claude-sonnet-5 | 96/100 | 1139 | $4.4384 | 84.284 | 0/20 | 20/20 | self_reported |
sqlite-fts | claude-sonnet-5 | 96/100 | 1142 | $4.5335 | 84.063 | 2/20 | 20/20 | self_reported |
crux | claude-sonnet-5 | 95/100 | 1263 | $4.0884 | 75.218 | 20/20 | 20/20 | self_reported |
rag-bm25 | claude-haiku-4-5 | 20/20 | 145 | $0.3267 | 137.931 | 6/20 | 5/20 | self_reported |
sqlite-fts | claude-haiku-4-5 | 20/20 | 146 | $0.2561 | 136.986 | 8/20 | 10/20 | self_reported |
vendor-native | claude-haiku-4-5 | 20/20 | 239 | $0.1804 | 83.682 | 5/20 | 20/20 | self_reported |
crux | claude-haiku-4-5 | 20/20 | 265 | $0.1300 | 75.472 | 20/20 | 20/20 | self_reported |
crux | claude-sonnet-4-6 | 15/15 | — | $0.7102 | — | — | — | self_reported |
vendor-native | claude-sonnet-4-6 | 15/15 | — | $0.8016 | — | — | — | self_reported |
vendor-native | haiku | 10/12 | — | — | — | — | — | self_reported |
compaction | claude-haiku-4-5 | 10/20 | 124 | $0.1752 | 80.645 | 0/20 | 20/20 | self_reported |
crux | haiku | 9/12 | — | — | — | — | — | self_reported |
crux | sonnet | 9/9 | — | — | — | — | — | self_reported |
vendor-native | sonnet | 9/9 | — | — | — | — | — | self_reported |
none | claude-haiku-4-5 | 5/20 | — | $0.7907 | — | 5/20 | 0/20 | self_reported |
none | claude-sonnet-4-6 | 2/15 | — | $1.2816 | — | — | — | self_reported |
none | claude-sonnet-5 | 0/100 | — | $5.4890 | — | 0/20 | 0/20 | self_reported |
random | synthetic | 0/100 | — | — | — | 0/20 | 0/20 | self_reported |
none | haiku | 0/12 | — | — | — | — | — | self_reported |
none | sonnet | 0/9 | — | — | — | — | — | self_reported |
Fingerprint
Section × backend (correct / n)
| Section | none | vendor-native | compaction | rag-bm25 | sqlite-fts | crux | oracle | random |
|---|---|---|---|---|---|---|---|---|
| S1Rederivable leak gate | 8/8 | 8/8 | 5/5 | 5/5 | 5/5 | 8/8 | 5/5 | 0/5 |
| S2Arbitrary decisions | 0/38 | 37/38 | 19/20 | 18/20 | 18/20 | 37/38 | 20/20 | 0/20 |
| S3Cross-session continuity | 0/23 | 22/23 | 19/20 | 19/20 | 19/20 | 22/23 | 20/20 | 0/20 |
| S4Causal / why-chains | 2/23 | 23/23 | 20/20 | 20/20 | 20/20 | 23/23 | 20/20 | 0/20 |
| S5Supersession control | 5/49 | 47/49 | 30/40 | 40/40 | 40/40 | 49/49 | 20/20 | 0/20 |
| S6Scale / needle | 0/23 | 22/23 | 19/20 | 19/20 | 19/20 | 17/23 | 20/20 | 0/20 |
| S8Provenance / trust provenance axis | 5/40 | 5/40 | 0/40 | 6/40 | 10/40 | 40/40 | 20/20 | 0/20 |
| S9Drift / longitudinal drift axis | 0/40 | 40/40 | 40/40 | 25/40 | 30/40 | 40/40 | 20/20 | 0/20 |
Fairness invariants: S1 no-leak · oracle ceilings · random floors.
Runs
Every run
| Section | Backend | Model | Correct | Δ vs vendor | Ctx tok | Latency | Runs |
|---|---|---|---|---|---|---|---|
| S1 | compaction | claude-sonnet-5 | 5/5 | 0 | 40 | 6723ms | 1 |
| S1 | crux | claude-sonnet-5 | 5/5 | — | 48 | 6019ms | 1 |
| S1 | none | claude-sonnet-5 | 5/5 | 0 | 0 | 8744ms | 1 |
| S1 | oracle | synthetic | 5/5 | — | 0 | — | 2 |
| S1 | rag-bm25 | claude-sonnet-5 | 5/5 | 0 | 51 | 7910ms | 1 |
| S1 | sqlite-fts | claude-sonnet-5 | 5/5 | 0 | 51 | 9370ms | 1 |
| S1 | vendor-native | claude-sonnet-5 | 5/5 | 0 | 42 | 7527ms | 1 |
| S1 | crux | claude-sonnet-4-6 | 3/3 | 0 | — | — | 1 |
| S1 | none | claude-sonnet-4-6 | 3/3 | 0 | — | — | 1 |
| S1 | vendor-native | claude-sonnet-4-6 | 3/3 | 0 | — | — | 1 |
| S1 | random | synthetic | 0/5 | — | 0 | — | 2 |
| S2 | oracle | synthetic | 20/20 | — | 0 | — | 4 |
| S2 | compaction | claude-sonnet-5 | 19/20 | 0 | 139 | 35707ms | 1 |
| S2 | crux | claude-sonnet-5 | 19/20 | — | 158 | 20991ms | 1 |
| S2 | vendor-native | claude-sonnet-5 | 19/20 | 0 | 141 | 31825ms | 1 |
| S2 | rag-bm25 | claude-sonnet-5 | 18/20 | -1 | 156 | 26163ms | 1 |
| S2 | sqlite-fts | claude-sonnet-5 | 18/20 | -1 | 156 | 29542ms | 1 |
| S2 | crux | claude-sonnet-4-6 | 6/6 | 0 | — | — | 1 |
| S2 | crux | haiku | 6/6 | 0 | — | — | 3 |
| S2 | crux | sonnet | 6/6 | 0 | — | — | 1 |
| S2 | vendor-native | claude-sonnet-4-6 | 6/6 | 0 | — | — | 1 |
| S2 | vendor-native | haiku | 6/6 | 0 | — | — | 3 |
| S2 | vendor-native | sonnet | 6/6 | 0 | — | — | 1 |
| S2 | none | claude-sonnet-4-6 | 0/6 | -6 | — | — | 1 |
| S2 | none | claude-sonnet-5 | 0/20 | -19 | 0 | 57205ms | 1 |
| S2 | none | haiku | 0/6 | -6 | — | — | 3 |
| S2 | none | sonnet | 0/6 | -6 | — | — | 1 |
| S2 | random | synthetic | 0/20 | — | 0 | — | 4 |
| S3 | oracle | synthetic | 20/20 | — | 0 | — | 2 |
| S3 | compaction | claude-sonnet-5 | 19/20 | 0 | 191 | 28436ms | 1 |
| S3 | crux | claude-sonnet-5 | 19/20 | — | 210 | 23061ms | 1 |
| S3 | rag-bm25 | claude-sonnet-5 | 19/20 | 0 | 213 | 22668ms | 1 |
| S3 | sqlite-fts | claude-sonnet-5 | 19/20 | 0 | 213 | 26659ms | 1 |
| S3 | vendor-native | claude-sonnet-5 | 19/20 | 0 | 193 | 29579ms | 1 |
| S3 | crux | claude-sonnet-4-6 | 3/3 | 0 | — | — | 1 |
| S3 | vendor-native | claude-sonnet-4-6 | 3/3 | 0 | — | — | 1 |
| S3 | none | claude-sonnet-4-6 | 0/3 | -3 | — | — | 1 |
| S3 | none | claude-sonnet-5 | 0/20 | -19 | 0 | 53832ms | 1 |
| S3 | random | synthetic | 0/20 | — | 0 | — | 2 |
| S4 | compaction | claude-sonnet-5 | 20/20 | 0 | 280 | 35936ms | 1 |
| S4 | crux | claude-sonnet-5 | 20/20 | — | 298 | 19710ms | 1 |
| S4 | oracle | synthetic | 20/20 | — | 0 | — | 2 |
| S4 | rag-bm25 | claude-sonnet-5 | 20/20 | 0 | 301 | 31662ms | 1 |
| S4 | sqlite-fts | claude-sonnet-5 | 20/20 | 0 | 302 | 31703ms | 1 |
| S4 | vendor-native | claude-sonnet-5 | 20/20 | 0 | 281 | 19747ms | 1 |
| S4 | crux | claude-sonnet-4-6 | 3/3 | 0 | — | — | 1 |
| S4 | vendor-native | claude-sonnet-4-6 | 3/3 | 0 | — | — | 1 |
| S4 | none | claude-sonnet-4-6 | 2/3 | -1 | — | — | 1 |
| S4 | none | claude-sonnet-5 | 0/20 | -20 | 0 | 65289ms | 1 |
| S4 | random | synthetic | 0/20 | — | 0 | — | 2 |
| S5 | compaction | claude-sonnet-5 | 20/20 | 0 | 124 | 30110ms | 1 |
| S5 | crux | claude-haiku-4-5 | 20/20 | 0 | 265 | 33724ms | 1 |
| S5 | crux | claude-sonnet-5 | 20/20 | — | 265 | 21129ms | 1 |
| S5 | oracle | synthetic | 20/20 | 0 | 0 | — | 4 |
| S5 | rag-bm25 | claude-haiku-4-5 | 20/20 | 0 | 145 | 66328ms | 1 |
| S5 | rag-bm25 | claude-sonnet-5 | 20/20 | 0 | 145 | 46887ms | 1 |
| S5 | sqlite-fts | claude-haiku-4-5 | 20/20 | 0 | 146 | 99178ms | 1 |
| S5 | sqlite-fts | claude-sonnet-5 | 20/20 | 0 | 146 | 59211ms | 1 |
| S5 | vendor-native | claude-haiku-4-5 | 20/20 | 0 | 239 | 80589ms | 1 |
| S5 | vendor-native | claude-sonnet-5 | 20/20 | 0 | 238 | 33444ms | 1 |
| S5 | compaction | claude-haiku-4-5 | 10/20 | -10 | 124 | 58923ms | 1 |
| S5 | none | claude-haiku-4-5 | 5/20 | -15 | 0 | 225354ms | 1 |
| S5 | crux | claude-sonnet-4-6 | 3/3 | 0 | — | — | 1 |
| S5 | crux | haiku | 3/3 | +2 | — | — | 3 |
| S5 | crux | sonnet | 3/3 | 0 | — | — | 1 |
| S5 | vendor-native | claude-sonnet-4-6 | 3/3 | 0 | — | — | 1 |
| S5 | vendor-native | sonnet | 3/3 | 0 | — | — | 1 |
| S5 | vendor-native | haiku | 1/3 | 0 | — | — | 3 |
| S5 | none | claude-sonnet-4-6 | 0/3 | -3 | — | — | 1 |
| S5 | none | claude-sonnet-5 | 0/20 | -20 | 0 | 83665ms | 1 |
| S5 | none | haiku | 0/3 | -1 | — | — | 3 |
| S5 | none | sonnet | 0/3 | -3 | — | — | 1 |
| S5 | random | synthetic | 0/20 | -20 | 0 | — | 4 |
| S6 | oracle | synthetic | 20/20 | — | 0 | — | 2 |
| S6 | compaction | claude-sonnet-5 | 19/20 | 0 | 186 | 29328ms | 1 |
| S6 | rag-bm25 | claude-sonnet-5 | 19/20 | 0 | 324 | 27086ms | 1 |
| S6 | sqlite-fts | claude-sonnet-5 | 19/20 | 0 | 325 | 28866ms | 1 |
| S6 | vendor-native | claude-sonnet-5 | 19/20 | 0 | 7575 | 29252ms | 1 |
| S6 | crux | claude-sonnet-5 | 17/20 | — | 332 | 29026ms | 1 |
| S6 | vendor-native | haiku | 3/3 | 0 | — | — | 1 |
| S6 | crux | haiku | 0/3 | -3 | — | — | 1 |
| S6 | none | claude-sonnet-5 | 0/20 | -19 | 0 | 49889ms | 1 |
| S6 | none | haiku | 0/3 | -3 | — | — | 1 |
| S6 | random | synthetic | 0/20 | — | 0 | — | 2 |
| S8 | crux | claude-haiku-4-5 | 20/20 | +15 | 490 | 45698ms | 1 |
| S8 | crux | claude-sonnet-5 | 20/20 | — | 490 | 32303ms | 1 |
| S8 | oracle | synthetic | 20/20 | +15 | 0 | — | 1 |
| S8 | sqlite-fts | claude-haiku-4-5 | 8/20 | +3 | 370 | 110102ms | 1 |
| S8 | rag-bm25 | claude-haiku-4-5 | 6/20 | +1 | 370 | 116938ms | 1 |
| S8 | none | claude-haiku-4-5 | 5/20 | 0 | 0 | 230592ms | 1 |
| S8 | vendor-native | claude-haiku-4-5 | 5/20 | 0 | 336 | 207163ms | 1 |
| S8 | sqlite-fts | claude-sonnet-5 | 2/20 | +2 | 370 | 111644ms | 1 |
| S8 | compaction | claude-haiku-4-5 | 0/20 | -5 | 333 | 178605ms | 1 |
| S8 | compaction | claude-sonnet-5 | 0/20 | 0 | 333 | 195531ms | 1 |
| S8 | none | claude-sonnet-5 | 0/20 | 0 | 0 | 64604ms | 1 |
| S8 | rag-bm25 | claude-sonnet-5 | 0/20 | 0 | 370 | 117564ms | 1 |
| S8 | random | synthetic | 0/20 | -5 | 0 | — | 1 |
| S8 | vendor-native | claude-sonnet-5 | 0/20 | 0 | 335 | 212482ms | 1 |
| S9 | compaction | claude-haiku-4-5 | 20/20 | 0 | 152 | 65678ms | 1 |
| S9 | compaction | claude-sonnet-5 | 20/20 | 0 | 152 | 16995ms | 1 |
| S9 | crux | claude-haiku-4-5 | 20/20 | 0 | 294 | 39303ms | 1 |
| S9 | crux | claude-sonnet-5 | 20/20 | 0 | 294 | 22398ms | 1 |
| S9 | oracle | synthetic | 20/20 | 0 | 0 | — | 1 |
| S9 | rag-bm25 | claude-sonnet-5 | 20/20 | 0 | 174 | 46339ms | 1 |
| S9 | sqlite-fts | claude-sonnet-5 | 20/20 | 0 | 175 | 38790ms | 1 |
| S9 | vendor-native | claude-haiku-4-5 | 20/20 | 0 | 858 | 113261ms | 1 |
| S9 | vendor-native | claude-sonnet-5 | 20/20 | 0 | 858 | 71761ms | 1 |
| S9 | sqlite-fts | claude-haiku-4-5 | 10/20 | -10 | 175 | 137292ms | 1 |
| S9 | rag-bm25 | claude-haiku-4-5 | 5/20 | -15 | 174 | 189822ms | 1 |
| S9 | none | claude-haiku-4-5 | 0/20 | -20 | 0 | 223428ms | 1 |
| S9 | none | claude-sonnet-5 | 0/20 | -20 | 0 | 40943ms | 1 |
| S9 | random | synthetic | 0/20 | -20 | 0 | — | 1 |
Sections
What each section measures
S1Rederivable leak gate
The answer is in a sandbox file any backend can read.
e.g. Internal port configured? (check the repo)
Win/loss: Everyone should tie `none`. A lift here means the backend LEAKED the gold — it fails the build. Excluded from the /100.
S2Arbitrary decisions
A non-rederivable codename / port / flag, knowable only from prior knowledge.
e.g. Internal port assigned (no derivable pattern)?
Win/loss: Memory beats cold. A dump and a retriever both supply the value.
S3Cross-session continuity
A prior session’s decision must govern the current task.
e.g. Continuing the prior session — which storage engine was chosen?
Win/loss: Memory beats cold; correctness only if the prior decision is respected.
S4Causal / why-chains
The rationale (“why X over Y”) exists only in recorded prior knowledge.
e.g. Why was cbor chosen over protobuf for the wire format?
Win/loss: Memory beats cold on rationale, not just values.
S5Supersession control
A fact was changed; only the CURRENT value is correct. The dump lists both, unresolved.
e.g. What is the CURRENT internal port (it was changed)?
Win/loss: Naive memory surfaces stale and fails; only a freshness-aware backend wins — decisively for weak models.
S6Scale / needle
A needle among N distractors. The dump is O(N) tokens; retrieval is O(k).
e.g. What internal port was recorded in the notes?
Win/loss: Same accuracy, a fraction of the token bill — until the dump breaks the window (see Scale).
S8Provenance / trust provenance axis
Two sources disagree; only the confidence tells you which to trust. Arbitrary values, so guessing = 50%. Scored separately — NOT in the /100.
e.g. Two sources disagree on the datacenter — use the more confident one. What is it?
Win/loss: Backends that carry confidence (crux) pick the trusted value; ones that drop it (vendor-native, rag-bm25) can only guess. This is where provenance earns its tokens.
S9Drift / longitudinal drift axis
Each key is updated K times over a session (v1 → … → vK); only vK is current. Scored separately — NOT in the /100.
e.g. The rate_limit was updated 6 times this session — what is its CURRENT value?
Win/loss: A backend that RESOLVES to current (crux, and retrievers that store the current value) tracks the latest cheaply; a raw accumulating dump (vendor-native) makes the model find the latest among the interleaved log and drifts — worse the longer the session runs.
Backends
The backends, and what each does better
none
Empty context — the floor.
observed: 12/236 (5%)
vendor-native
A full CLAUDE.md-style rules dump, S5 history unresolved. The honest baseline; O(N) tokens.
observed: 196/236 (83%) · 11054 ctx tok
compaction
Current-value-only dump, routine notes elided. “Summarise your CLAUDE.md.”
observed: 147/200 (74%) · 2014 ctx tok
rag-bm25
In-process BM25 retrieval, top-k per probe. No infra; resolves S5; O(k) tokens.
observed: 147/200 (74%) · 2372 ctx tok
sqlite-fts
Stdlib sqlite3 FTS5 retrieval — the neutral worked example to copy.
observed: 156/200 (78%) · 2378 ctx tok
crux
The Crux daemon: freshness-resolved / retrieved slice. One row among the rest.
observed: 228/236 (97%) · 3096 ctx tok
Add yours: implement (plant, assemble) in adapters.py per BACKENDS.md, or POST results to /api/context/submit with your gold_sha256.