Community Leaderboard v1.5
Agent Effectiveness Metric Standard
The public leaderboard currently scores runs with the v1.5 community rubric: quality, context, continuity, judge-assisted reasoning, and a soft cost-efficiency factor.
Composite
The Crux Score
Formula
Cx = S_gate × Q_combined × T_human_minutes × (1 / (1 + N_corrections)) × C_efficiency
Where:
S_gate ∈ {0, 1} — safety hard gate
Q_combined = (3·Q_info + 2·Q_context + 2·Q_continuity + 2·Q_reasoning) / active_weight_sum
T_human_minutes = T_human / 60
N_corrections = user correction count
C_efficiency = 1 / (1 + C_tokens_usd * 2)
Judge-assisted reasoning inputs (v1.5):
Q_reasoning = avg(R_judge, R_conflict_resolution, R_noise_resistance)Top Floor
Campaign Scoring
Per-Floor Score
floor_score = (completion / duration_min) / turns * 1000
Where:
completion ∈ [0, 1] — objective completion for this floor
duration_min — wall-clock time for this floor (minutes)
turns — LLM turns used on this floor
Example: 100% completion in 0.8 min with 17 turns
= (1.0 / 0.8) / 17 * 1000 = 73.5Campaign Composite
campaign_score = sum(floor_score) for all cleared floors
Each floor is scored independently at submission time.
Adding a new floor never recalculates previous floor scores.
The campaign score only grows as more floors are cleared.The per-floor formula rewards quality and efficiency independently for each floor. A fast floor with fewer turns scores higher than a slow floor with many turns, even at the same completion rate. The campaign composite is additive: clearing more floors always increases the total score. This means a campaign with 10 floors at moderate efficiency will outscore a campaign with 3 perfect floors, reflecting both breadth and depth of progress.
| Floor | Completion | Time | Turns | Floor Score |
|---|---|---|---|---|
| 1 | 100% | 1.98m | 23 | 22.0 |
| 2 | 100% | 0.81m | 17 | 72.6 |
| 3 | 100% | 2.12m | 13 | 36.3 |
| Campaign Total | 130.9 | |||
Example from Sonnet 4.6 T2 on Floors 1 to 3. Floor 2 scored highest (72.6) because it was completed fastest with the fewest turns. Floor 1 scored lowest (22.0) despite 100% completion because it took more turns.
Top Floor
Per-Floor Diagnostics
Each floor run produces diagnostic scores across five dimensions. All dimensions are scored 0 to 1.
| Dimension | What it measures | Key sub-metrics |
|---|---|---|
| Retrieval | Signal vs noise separation in document search | Triage precision, recall, search efficiency, query refinement rate, time to first signal |
| Reasoning | Quality of logical deduction from evidence | Deduction accuracy, false lead resistance, synthesis events, hypothesis formation |
| Problem Solving | Strategic approach to objectives | Dependency-aware ordering (Kendall tau), code challenge pass rate, elevator key derivation |
| Efficiency | Resource usage and tool effectiveness | Turns per objective, tokens per objective, idle turn ratio, tool call success rate |
| Memory | Continuity across memory wipes | Pre-wipe preparation, recovery rate, cross-floor recall, wipe detection speed |
Top Floor
Configuration Recommendations
The diagnostic system maps performance patterns to specific VaultCrux/MemoryCrux configuration changes. Recommendations are rule-based and include the specific feature flag or parameter to adjust.
| Pattern | Recommendation | Config |
|---|---|---|
| Triage precision < 40% | Enable cross-encoder reranking | FEATURE_CROSS_ENCODER_RERANK |
| Many searches, low refinement | Enable HyDE query expansion | FEATURE_DQP_HYDE_RETRIEVAL |
| Slow first signal (>50% turns) | Increase retrieval topK | RETRIEVAL_LIMIT: 8 → 20 |
| Low recall on temporal corpus | Enable temporal surface | FEATURE_SURFACE_VERSION_CHAIN |
| Wipe recovery < 30% on T1 | Switch to T2 (persistent memory) | Arm change |
| Context-stuffing > 2x budget | Switch to tool-mediated retrieval | Arm change |
Foundations
Design Principles
1. Time is the anchor
Every composite metric resolves to a time unit. Time is universal, intuitive, and what humans optimise for.
2. Safety is a gate, not a gradient
An unsafe session scores zero. There is no partial credit for "almost safe."
3. Layers are independent
Pipeline, LLM, and agent metrics are measured and reported separately. Composites combine layers explicitly.
4. Immutable definitions
Once published at v1.0, a formula and unit cannot change. Deprecation with a replacement pointer is allowed; redefinition is not.
5. Versioned and disclosed
Every public run must disclose its metrics_version. v1.5 community runs include an LLM judge signal and should not be described as fully objective scoring.
Reporting
Null Handling
Dimensions that cannot be measured for a given run are recorded as null. Derived metrics that depend on null fundamentals are also null. The Crux Score uses only non-null components in Q_combined — the denominator adjusts to the sum of weights for non-null components.
Changelog
Version History
| Version | Date | Changes |
|---|---|---|
| 1.0 | 2026-03-26 | Initial publication of the canonical core metric set. |
| 1.1 | 2026-03-29 | +5 fundamentals (I6–I9, K4), +2 derived (Q5, V4). Memory benchmark coverage gaps. |
| 1.2 | 2026-03-31 | +2 fundamentals (I10–I11), +1 derived (Q6). Proposition-level partial credit. |
| 1.4 | 2026-04-06 | Community leaderboard added the soft cost-efficiency multiplier to the composite. |
| 1.5 | 2026-04-11 | Community leaderboard added Q_reasoning from judge/conflict/noise signals and now reports metrics_version: 1.5. |
Context
Relationship to Established Benchmarks
| Established Metric | ScoreCrux Equivalent | Difference |
|---|---|---|
| METR time horizon | T_human × pass_rate | METR = capability ceiling; ScoreCrux = effectiveness per session |
| SWE-bench resolved | S_gate × (R_decision ≥ threshold) | Binary pass/fail vs. quality decomposition |
| tau-bench pass^k | Cx mean ± std | Same consistency concept, different formula |
| CLEAR Cost/Latency/Efficacy | E1 / T1+T2 / Q1 | Direct mappings with finer decomposition |