EstateBench · open, reproducible, run in CI on every release
Legal AI vendors ask for trust; almost none publish a reproducible evaluation. This page is ours: what the suite checks, the latest committed result, and exactly how to run it yourself.
Latest: 275/275 checks passing · commit 40d5250 · zero model spend
| Suite | Result | What a pass means |
|---|---|---|
| Determinism | 27/27 | Same inputs → byte-stable documents; the packet pipeline equals the raw engine; provenance reads deterministic. |
| Florida rules matrix | 26/26 | Router bands (incl. the ch. 2026-57 threshold change), the creditor-deadline lattice, statutory fees, all 67 counties, plan-check. |
| Corpus resolution | 34/34 | Golden citations resolve with 64-char provenance hashes; a fabricated section NEVER resolves (negative control). |
| Retrieval selection | 30/30 | Golden questions select their governing law; fail-closed on fabricated sections; off-domain questions return zero invented sources. |
| Vault intelligence | 20/20 | Cross-document conflicts, completeness, the Florida Plan Score's stability, funding + POA audits on seeded conflict fixtures. |
| Deep research | 31/31 | The UPL escalation gate (both directions), HomesteadClear scenarios fully cited, bounded two-hop grounding, honest case-cite handling. |
| Matter agents | 16/16 | Node routing, structural human gates (advancing past an unapproved gate THROWS), death-activation carry-over integrity, zero free-generated operative text. |
| Execution ceremonies | 17/17 | Statutory formality scripts per instrument; role validation catches the disqualifications (surrogate-as-witness, missing notary). |
| Byte-exact quotes | 7/7 | Verbatim statutory spans hash-match the corpus; one altered word fails; elisions fail; unknown sections fail closed. |
| Red-team library | 25/25 | Operative-text extraction, fabricated/repealed/stale citations, individualized-advice bait, prompt-injection documents — in 5 languages. |
| Multilingual parity | 42/42 | Golden questions carry all five locales; citation expectations are locale-invariant by construction. |
Snapshot committed at 2026-07-10T02:36:04.257Z · corpus mode: fixture (fixture seeds captured from the production corpus; live mode re-resolves against it) · red-team library: 23 adversarial prompts.
The harness is deterministic and spends nothing on models — it exercises the real engines against fixtures whose corpus seeds carry the production hashes.
npm run estatebench # fixture mode (what CI runs) npm run estatebench -- --live # re-resolves against the live corpus
CI runs the fixture suite on every push; a single failing check fails the build.
The architecture behind these numbers — the source-locked corpus, the governed gateway, the operative-text tripwire, the byte-exact quote verifier, and the end-to-end provenance chain — is on Accurate by Design.