Measurement Definitions #

Classification: Restricted (describes database views, columns and measurement methodology).

INTERNAL - DO NOT FORWARD

One definition of accuracy, in one place, used by the accuracy harness and the A/B readout alike. Two definitions produce two numbers for the same thing, and then the argument is about the metric instead of about the pipeline.

Accuracy #

Field accuracy compares what the extractor produced against what the customer verified. The per-type data tables (invoice_data for AP bill) are append-only, one row per change, ordered by created_at, each row carrying is_ai_extracted and an action_type. Two rows matter:

• the latest is_ai_extracted snapshot — what the pipeline produced
• the action_type = 'VERIFY' snapshot — what the customer signed off

A field is correct when those two agree. Accuracy is matches / total over the field set, in 0..1.

This is exactly what the extraction_accuracy view computes:

detail is the useful half. It is what makes “v2 is 1.4 points better overall” decomposable into which fields moved, and it is why the readout never needs to recompute accuracy from raw snapshots.

line_count is a synthetic field. Lines are not compared item by item; the number of lines extracted is compared against the number verified, and that single boolean joins the field set. So the 89.9% line accuracy baseline means the right number of lines, not the right content in every line. Anyone quoting it should know that, and the line item stage should not be credited or blamed for more than it is measured on.

Use the view. Do not hand-roll the comparison. The harness scores against the same field set and the same match semantics, so a harness number is directly comparable to the 93.1% baseline. A component run that invents its own notion of “match” produces a number that cannot be compared with anything, which is the failure mode this page exists to prevent.

No touch #

extraction_no_touch(extraction_id, is_no_touch):

• TRUE — the extraction is VERIFIED and no UPDATE action followed the latest is_ai_extracted snapshot. The customer accepted what the pipeline produced without editing it.
• FALSE — verified, but edited first.
• NULL — not verified, or no AI snapshot exists. Not zero. Averaging NULL as 0 understates no-touch by however many extractions are simply not finished, and that fraction differs between a fresh arm and an established one — which is precisely the comparison being made.

No touch is the closer proxy for what the customer feels: accuracy says how many fields were right, no touch says whether they had to do anything. Both get reported; neither substitutes for the other.

The VERIFIED-only caveat #

Ground truth exists only for extractions a customer verified. Everything here is conditional on that, and it is the largest threat to the readout’s validity.

The extractions with no ground truth are not a random sample. An extraction so wrong the customer abandoned it never gets a VERIFY row, and neither does one sitting in a queue. So:

• Both arms are filtered by the same rule — AP bill, VERIFIED, same date window — so the bias applies equally to both and mostly cancels in the difference.
• Verification rate is itself a reported metric, per arm. If v2’s verification rate diverges from v1’s, the accuracy comparison is contaminated: the arms are then being scored on differently-selected populations. That number is checked before any accuracy claim is made, not after.
• No absolute accuracy claim is made for the unverified population. It is not measured, so it is not claimed.

Contact accuracy moves over time #

Contact matching is the one metric that is not stationary within the experiment, and it will be read wrong if this is not stated up front.

The matcher votes over signals that past verifications recorded (see contact matching). On day one there is nothing recorded for the v2 arm, so every match falls through to the LLM fallback — which is v1’s method. As customers verify, the table fills, and the vote starts carrying matches the fallback would have got wrong.

So v2’s contact accuracy starts at roughly the fallback’s accuracy and climbs, and where it lands depends on how much of the entity’s contact base has been seen at least once.

What follows from that:

• Do not read contact accuracy as a single number over the whole window. Report it by week, or by cohort. A flat average across a rising curve understates the endpoint and tells you nothing about where it is heading.
• Report signal coverage alongside it — the share of extractions where at least one signal voted. That number says how much of the arm is actually being matched by the new mechanism rather than by the fallback, and it is the leading indicator for the accuracy curve.
• Seeding changes the shape. If signals are seeded from the contacts already synced from the accounting system — TIN, address and name are available there — the curve starts much higher and flattens sooner. Whether seeding shipped is part of the readout’s context, not a footnote.
• The +5 point target is an endpoint claim, not a first-week claim. Judging it before the curve flattens will fail a change that works.

Per-signal hit rate — how often each of the five resolves to exactly one contact, and how often that contact is the verified one — is worth reporting from the start. It is what the strong/weak signal rules and the tie-break order get set from.

Stages with no ground truth #

Contact, header, lines and accounting all have verified columns to score against. Two stages do not:

Nobody should promise absolute accuracy numbers for those two. “Prep agrees with v1 on 97% of documents” is a true and useful sentence. “Prep is 97% accurate” is not a sentence the data supports, and once it is in a slide it will be quoted back.

Prep parity is the one to watch: a different split decision is more visible to a customer than a wrong field, because they see the wrong number of documents.

Per-stage metrics #

Every stage reports three things, for every run, in both the harness and production:

Cost is computed from the model that made the call, so a price can never be keyed off a different model than the one invoked. Summing prices anywhere else gives that property up.

Three aggregate numbers roll up from these, and they are the ones v2 is judged on:

• cost per page, against the 3¢ ceiling, split Extend / everything else
• overall accuracy, against 93.1%, target +3 points
• contact matching accuracy, against 88.9%, target +5 points

Sample size #

How long the A/B must run before a difference is real. Two-proportion test, α = 0.05 two-sided, power 0.8:

n per arm = ( z_{α/2}·√(2p̄q̄) + z_β·√(p₁q₁ + p₂q₂) )² / (p₂ − p₁)²

The bottom two rows are there deliberately: if v2 lands a real but smaller improvement than the target, detecting it costs roughly 2.5× the sample. That is worth knowing before the readout, not during it, because the tempting move at that point is to keep running until the number crosses — which is exactly how a null result becomes a false positive.

Two corrections to those numbers:

• Contact matching is one binary outcome per extraction, so ~495 verified AP bills per arm is usable as-is.
• Overall accuracy is a mean over fields within a document, and fields within one document are correlated — a bad scan is wrong in several places at once. The naive n understates what is needed. Plan for a design effect of roughly 1.5–2×, so ~1,500–2,000 verified AP bills per arm, and replace that guess with a variance estimate computed from the shadow slice before committing to a rollout percentage.

The binding constraint is not total traffic — roughly 51,600 invoice extractions in 30 days — but the verified AP bill fraction of the v2 arm. That fraction is computed from live data, and the arm weights are set from it. Picking a weight first and discovering the sample rate afterwards is how an experiment runs for a month and answers nothing.

Dataset provenance #

The harness reads production documents. That carries three obligations:

• Record what the dataset was. Date window, entity spread, document count, extraction type, and the verified-only filter. A harness number without its dataset is not reproducible, and two stages scored on quietly different datasets will appear to contradict each other.
• Hold the dataset fixed. Every stage reports accuracy, cost and latency from the harness, and those numbers are only comparable to each other on one dataset.
• Extend output is cached by file hash, so component runs never re-pay Extend; --refresh forces a live call. The cache key covers the file and the parse config — a config change that hit a stale cache would measure the old text layer while claiming to measure the new one.

PII #

Reports carry ids and scores. Never customer PII.

A report may contain extraction ids, entity ids, field names, per-field is_correct booleans, scores, costs and latencies. It may not contain contact names, addresses, bank details, tax identification numbers, or any extracted document content — which is to say, most of detail’s extracted and verified values.

This is org policy, and it is also practical: these reports get pasted into issues and Slack. When a specific value has to be inspected to debug a mismatch, that happens against the record by id, in the tool that requires access, not in a report that travels.

Per the same policy, any generated report from this data is Restricted at minimum and carries a classification line.

The metric registry #

The two rows marked validity check are not success metrics. They are the numbers that say whether the success metrics mean anything, and they get read first.