Held-Out Numbers You Can Audit
Dated obligation families—renewal, termination notice, payment terms—score 1.00 precision and 0.95 recall on the enforced half of an adversarial held-out corpus: real-shaped, messy documents the extractor never saw during development. The harder half, which we do not yet pass, reads 0.31 recall, and we publish that too. Every push runs the suite against fixed floors (F1 0.97, recall 0.85) and a build below them fails; those floors are set by hand. For the grounded LLM lane that reads standing commitments we publish no accuracy numbers at all — see below for why, and for what the code enforces instead.
Why we report held-out numbers
An earlier version of our evaluation used a self-referential fixture suite. It showed a perfect 1.00 recall—which collapsed to 0.32 when measured on an independent corpus of messy, real-shaped documents. We rebuilt extraction against that reality. That is why every number on this page comes from documents the extractor never saw during development, and why the numbers are not a claim about every customer PDF.
- Adversarial held-out corpus: real-shaped, messy documents excluded from development
- Dated families measured at 1.00 precision / 0.95 recall on the enforced half of that corpus
- The advisory half—documents we still get wrong—reads 0.31 recall, and we publish it
- Fixed CI floors (F1 0.97, recall 0.85) run on every push; a build below them fails
- The floors are set by hand. The ratchet is manual: a document graduates into the enforced corpus only once it passes
- Standing commitments: no published precision or recall—that lane is not measured on the push gate
- Confidence calibration verified across bands
| Metric | Value |
|---|---|
| Dated families — precision | 1.00 (enforced held-out) |
| Dated families — recall | 0.95 (enforced held-out) |
| Dated families — advisory corpus recall | 0.31 (published on purpose) |
| Standing commitments | Not published |
| Automated tests | ≈3,800 |
| CI accuracy floors | F1 0.97 / recall 0.85 (build fails below) |
Six pillars of extraction quality
From held-out precision and recall to CI enforcement and human review—every layer is designed so you can trust what ClauseMinds extracts, and check the claim yourself.
Precision on dated families
Renewal, termination notice, and payment terms score 1.00 precision on the adversarial held-out corpus—no spurious dated obligations against those documents. In production, low-confidence items route to review instead of silently passing.
Recall on dated families
0.95 recall on the enforced half of the same held-out corpus: some obligations in messy documents are still missed, and we say so. The advisory half—the documents we do not yet pass—reads 0.31, and we publish that number rather than bury it.
Standing commitments lane
We publish no accuracy figures for the grounded LLM read of confidentiality, exclusivity, IP, and similar standing terms. It needs a live model key, so it cannot run on the push gate, and we will not quote a number we cannot reproduce on demand. What is enforced in code: every item must quote its source clause or it is dropped, and nothing from this lane is auto-accepted.
Multi-Format Coverage
PDF, DOCX, TXT, HTML, XLSX, and scanned images with OCR. Native text and OCR fallback both evaluated.
Human Review Safety Net
Confidence routing sends low-confidence extractions to human review before tracking. No obligation is finalized without explicit approval.
CI-Enforced Floors
Evaluation gates run in CI on every push against fixed floors—F1 0.97 and held-out recall 0.85. Drop below either and the build fails. The floors are numbers we set by hand, not a self-raising ratchet. Roughly 3,800 automated tests run alongside them.
Reviewers are the live ground truth
Held-out numbers say how extraction performed before release; production tells us how it performs on your documents. Every reviewer accept, edit, and reject decision feeds the accuracy loop: rejection rates proxy precision, missed-obligation reports proxy recall, and drift is detected window-over-window. Human review remains the final gate—no obligation is tracked without approval for anything uncertain.
- Reviewer accept/edit/reject decisions are live ground truth
- Rejection rates proxy precision; missed-obligation reports proxy recall
- Drift detected window-over-window, not discovered by accident
- Low-confidence extractions always route to review
- Audit trail for every review decision
- medium
Renewal Notice
Acme MSA
- low
Termination Period
TechCo License
- medium
Payment Term
DataInc SaaS
Accuracy claims you can check
1.00 precision / 0.95 recall on dated obligations against the enforced held-out corpus, 0.31 recall on the advisory corpus we still fail, fixed CI floors that fail the build, and human review when confidence is thin. Held-out numbers, not tuned-fixture numbers — and no number at all for the lane we have not measured.