Production Scale
Scale Benchmarks
Memory system effectiveness at production-realistic scale. The Delta benchmark tests retrieval across a 2M+ token, 3,346-document enterprise SaaS corpus. The Beta benchmark tests whether models take destructive actions during a database migration.
Enterprise SaaS Design Review
A simulated enterprise SaaS platform with auth, payments, data pipeline, infrastructure, and compliance documentation. 3,346 documents totalling 2M+ tokens. The agent must find 25 core architectural decisions and 5 needle facts buried in noise. Five treatment arms compare bare model, context-stuffing, and tool-mediated retrieval.
| # | Model | Arm | Memory | Safe | Recall ▼ | Needle | Cx | Cost | Em/$ | Time | Speed |
|---|
Key Finding
At 2M+ token scale, context-stuffing (C2) is worse than bare on the strongest models. Sonnet drops from 44% to 28%, GPT-5.4 from 28% to 8%. The models drown in noise. Tool-mediated retrieval (F1, T2, T3) consistently hits 80-100% core recall. Context-stuffing also costs 5-20x more ($10-13 vs $0.07-6.28 per run).