Extraction Accuracy
These figures come from our internal evaluation harness, not from a third-party audit of live customer contracts. They are measured on held-out documents the extractor never saw during development. That corpus is one half of a split: documents we currently pass are the enforced set the numbers below describe, and the harder ones we do not yet pass are kept in a separate advisory set, where recall is 0.31. We publish both, because the second number is the one that predicts messy paper. Both corpora are internally authored, not customer contracts. Every push runs the suite against fixed floors (F1 0.97, recall 0.85) and a build below them fails; those floors are set by hand, not auto-ratcheted. Low-confidence candidates still require human review before anything is tracked in the product.
Last validated: (run python -m app.eval --held-out --fixtures-dir app/eval/fixtures from the backend to reproduce)
Why we report held-out numbers
Early in development, our own regression suite scored a perfect 1.00 recall. Then we built an independent, deliberately messy corpus of real-shaped documents the extractor had never seen — and recall collapsed to 0.32. The suite had been measuring how well the extractor matched its own fixtures, not how well it read contracts. We rebuilt extraction against that corpus until it reached precision 1.00 / recall 0.95, and made that corpus — not the fixtures — the one CI gates against. That is the number reported here, and it is the only kind of number we will report: measured on documents the system did not train toward.
Dated obligations — held-out adversarial corpus
Renewal clauses, termination notice periods, and payment terms are extracted by deterministic rules. This slice does not prove perfect accuracy on every contract in the wild; it measures the engine on adversarial documents held out of development.
| Metric | Value |
|---|---|
| Obligation types | renewal, termination_notice, payment_term |
| Precision | 1.00 on the held-out adversarial corpus |
| Recall | 0.95 on the same corpus |
| CI enforcement | every push runs the suite against fixed floors (F1 0.97, recall 0.85); a build below them fails. The floors are set by hand, not raised automatically — the ratchet is manual: a document graduates from the advisory corpus into the enforced one only once it passes cleanly |
| Confidence routing | low-confidence candidates are routed to human review; nothing is tracked without a source clause |
Standing commitments — no published numbers
Standing commitments (confidentiality, non-use, exclusivity, IP ownership, transfer/insurance/subcontracting limits) are read by a grounded LLM lane. We do not publish precision or recall for this lane. We measured it once, by hand, when the lane shipped. The code has changed since, and the nightly job that would re-measure it has not yet produced a successful run — so we will not quote a number we cannot reproduce on demand. What we can state is what the code enforces on every document, which is below.
| Metric | Value |
|---|---|
| Accuracy figures | not published — see above |
| Grounding rule | every item must quote its exact source clause; if the model cannot point at the text, the item is dropped before it reaches you |
| Auto-accept | none. Nothing from this lane is accepted without a human decision, at any confidence |
| Gate posture | precision-first: the lane is tuned to say less rather than risk asserting something unsupported |
| What we do publish | the dated-obligation numbers above, because those run on every push against a floor that fails the build |
Production accuracy loop
Benchmarks age; production does not. Reviewer accept, edit, and reject decisions are used as live ground truth: rejection rates proxy precision, missed-obligation reports proxy recall, and both are watched window-over-window with drift detection. Corrections feed back into the evaluation gate, so measured accuracy compounds with usage instead of decaying quietly.
Scope
Every number on this page applies to dated obligations only: renewal clauses, termination notice periods, and payment terms. Nothing here is a measurement of the standing-commitment lane. Supported ingest formats include PDF, DOCX, TXT, HTML, XLSX, and images. OCR is used for scanned pages; confidence is often lower on poor scans, which pushes more work to review.
Methodology
We label expected extractions per document, run the same extractor the product uses, and compute precision and recall by obligation type. Held-out documents are excluded from development so they measure generalization, not memorization. Adversarial cases cover ambiguous wording, nested clauses, amendments that change terms, and lookalike text. The standing-commitment lane is not part of this harness — it needs a live model key, so it cannot run on the push gate, and we treat its accuracy as unmeasured until it does. For implementation detail, see backend/app/eval/README.md in the repository.