Code-Minimalism Bench · CMB-v1
Code Minimalism
How much less code does an agent write when a minimalism discipline is in its context — and what did the current harness actually validate? A replay and extension of the MIT-licensed Ponytail agentic benchmark: their numbers published next to ours, with the correctness and reproduction gaps disclosed explicitly.
Results
Pooled deltas vs baseline — 12 real-repo feature prompts
Two discipline arms measured separately: the Ponytail plugin (upstream's, replayed by us on frontier models) and the Crux code-minimalism profile. Negative numbers are reductions vs an undisciplined baseline on the same tasks.
| Discipline arm | Model | n | Code (LOC) | Tokens | Cost* | Time | Validation | Measured by |
|---|---|---|---|---|---|---|---|---|
| Ponytail plugin upstream published | claude-haiku-4-5 | 4 | −54% | −22% | −20% | −27% | 20/20 safety (upstream) | upstream |
| Ponytail plugin v4.8.4, our replay | claude-sonnet-4-6 | 2 | −48.0% | −38.6% | −30.0% | −34.7% | non-empty diff only | us |
| Ponytail plugin v4.8.4, our replay | claude-fable-5 | 2 | −57.7% | −18.7% | −26.9% | −36.3% | non-empty diff only | us |
| Ponytail plugin v4.8.4, our replay | claude-opus-4-8 | 2 | −69.7% | −49.2% | −46.7% | −55.4% | non-empty diff only | us |
| Crux code-minimalism wizard profile, paired baseline | claude-fable-5 | 2 | −61.0% | −24.1% | −29.0% | −32.4% | non-empty diff only | us |
| Crux code-minimalism wizard profile, paired baseline | claude-opus-4-8 | 2 | −59.9% | −38.2% | −35.4% | −34.6% | non-empty diff only | us |
* Modelled list price from the Claude Code CLI's own accounting — subscription runs bill $0. Upstream row reproduced from their published results, attributed, not re-measured.
Reading it
What we think this shows — and what it doesn't
A real code-volume signal; correctness unmeasured
All 48 Crux-profile cells produced non-empty diffs. The pinned harness did not execute those patches or their tests, so CMB-v1 does not establish functional parity.
A code-volume effect first
Tokens are the weakest, noisiest axis: the discipline spends read tokens to avoid written lines, and on individual tasks token use went UP while LOC fell. Buy it for less code, not for token savings.
Stronger models over-build more
Pooled baseline grew from 1566 LOC (Sonnet) to 2242 (Opus) on identical tasks, while disciplined output converges (~680–815). The deltas grow with model tier.
No consistent winner between the two arms
The Crux profile was ahead on Fable, the Ponytail plugin on Opus. Same idea, within-band difference — read it that way.
Honest limitations
Read before quoting
- The replay harness historical `correct=1` field means only that git diff LOC was greater than zero; it does not execute the generated feature or tests. At least one timed-out Opus baseline still met that signal. Our rows relabel it and make no correctness claim.
- n=2 per cell in our runs (upstream used n=4). Directional, not tight. Baselines drifted ~17% day-to-day on Opus — our profile runs use same-day paired baselines, and so should any rerun.
- The effect is task-shaped: it concentrates in over-build-prone tasks (date-picker: 350 → 9 LOC on Opus) and is near zero on already-minimal tasks. The pooled number is not a per-task promise.
- Token counts include Claude Code's background haiku model (the CLI's own accounting; identical across arms). Cost is modelled list price, not billed spend.
- Not replayed: upstream's adversarial safety tier, and the caveman / yagni-oneliner control arms against the Crux profile. Upstream's own data shows terseness alone does NOT produce the effect (+7% tokens).
- LOC counts comments (upstream's metric, kept for comparability).
Reproduction status
Reproduce after the evidence hardening lands
The upstream harness, corpus, profile text and derived per-run records are pinned. The exact portable custom-arm patch and compact raw patch/CLI evidence are not yet on the default branch, so this is not currently a one-command reproduction. The retained records are committed in public-data/code-minimalism.
# harness (upstream, pinned) + corpus (pinned) git clone https://github.com/DietrichGebert/ponytail && git -C ponytail checkout 14a0d7954 git clone https://github.com/tiangolo/full-stack-fastapi-template git -C full-stack-fastapi-template checkout cd83fc1 # two archived, arm-symmetric harness patches: # a. MODELS["fable"] = "claude-fable-5" (run.py — Fable absent upstream) # b. token metric = modelUsage-sum (top-level usage is a partial slice) # REPRODUCIBILITY GAP: the exact third patch that added the custom Crux arm is # not yet archived on the default branch. The command below documents the # original invocation but is not a one-command reproduction until that patch # and CRUX_MINIMALISM_ARM_FILE support are merged. # Crux arm text = the code-minimalism profile body, verbatim (strip frontmatter): # github.com/CueCrux/Crux → crates/crux-config-wizard/profiles/code-minimalism.md TASKS12=tmpl-fe-datepicker,tmpl-fe-colorpicker,tmpl-fe-command,tmpl-fe-dropzone,\ tmpl-fe-wizard,tmpl-fe-rating,tmpl-be-duplicate,tmpl-be-search,tmpl-be-count,\ tmpl-be-archive,tmpl-be-bulkdelete,tmpl-be-csv PONYTAIL_TMPL=$PWD/full-stack-fastapi-template \ python3 ponytail/benchmarks/agentic/run.py \ --task "$TASKS12" --arms baseline,crux-minimalism --models fable --runs 2 --workers 4
Lineage: the ladder pattern and the harness are Ponytail's (MIT, credited). The Crux profile text is our own implementation of the idea; the rows above measure each separately. How the Crux daemon ships the discipline: cuecrux.com/code-minimalism.