Context-Dependence Benchmark
Context
How much does an agent task depend on carried context — and how well does your memory/context backend supply it? Fingerprint any backend across sections by knowledge type, with published negative controls, a /100 composite, and a first-class context-token bill. Backend-agnostic: bring your own backend and your own model.
Leaderboard
The /100 composite
Summed over the scored sections (S2–S6, 20 probes each), the S1 leak-gate excluded. The point is the fingerprint below, not one number — but this is the at-a-glance summary per backend, with the token bill it cost to supply the context.
| Backend | Model | Score | Context tokens | $ cost | correct / 1k-ctx | Provenance | Drift | Attribution |
|---|---|---|---|---|---|---|---|---|
| No scored runs yet. | ||||||||
Fingerprint
Section × backend (correct / n)
| Section |
|---|
Fairness invariants: S1 no-leak · oracle ceilings · random floors.
Runs
Every run
| Section | Backend | Model | Correct | Δ vs vendor | Ctx tok | Latency | Runs |
|---|---|---|---|---|---|---|---|
| No runs published yet. | |||||||
Sections
What each section measures
S1Rederivable leak gate
The answer is in a sandbox file any backend can read.
e.g. Internal port configured? (check the repo)
Win/loss: Everyone should tie `none`. A lift here means the backend LEAKED the gold — it fails the build. Excluded from the /100.
S2Arbitrary decisions
A non-rederivable codename / port / flag, knowable only from prior knowledge.
e.g. Internal port assigned (no derivable pattern)?
Win/loss: Memory beats cold. A dump and a retriever both supply the value.
S3Cross-session continuity
A prior session’s decision must govern the current task.
e.g. Continuing the prior session — which storage engine was chosen?
Win/loss: Memory beats cold; correctness only if the prior decision is respected.
S4Causal / why-chains
The rationale (“why X over Y”) exists only in recorded prior knowledge.
e.g. Why was cbor chosen over protobuf for the wire format?
Win/loss: Memory beats cold on rationale, not just values.
S5Supersession control
A fact was changed; only the CURRENT value is correct. The dump lists both, unresolved.
e.g. What is the CURRENT internal port (it was changed)?
Win/loss: Naive memory surfaces stale and fails; only a freshness-aware backend wins — decisively for weak models.
S6Scale / needle
A needle among N distractors. The dump is O(N) tokens; retrieval is O(k).
e.g. What internal port was recorded in the notes?
Win/loss: Same accuracy, a fraction of the token bill — until the dump breaks the window (see Scale).
S8Provenance / trust provenance axis
Two sources disagree; only the confidence tells you which to trust. Arbitrary values, so guessing = 50%. Scored separately — NOT in the /100.
e.g. Two sources disagree on the datacenter — use the more confident one. What is it?
Win/loss: Backends that carry confidence (crux) pick the trusted value; ones that drop it (vendor-native, rag-bm25) can only guess. This is where provenance earns its tokens.
S9Drift / longitudinal drift axis
Each key is updated K times over a session (v1 → … → vK); only vK is current. Scored separately — NOT in the /100.
e.g. The rate_limit was updated 6 times this session — what is its CURRENT value?
Win/loss: A backend that RESOLVES to current (crux, and retrievers that store the current value) tracks the latest cheaply; a raw accumulating dump (vendor-native) makes the model find the latest among the interleaved log and drifts — worse the longer the session runs.
Backends
The backends, and what each does better
Add yours: implement (plant, assemble) in adapters.py per BACKENDS.md, or POST results to /api/context/submit with your gold_sha256.