Routing & A/B Assignment #

Classification: Internal-Only

How an extraction gets assigned to an arm, why the assignment is per extraction rather than per entity, and what makes the resulting comparison readable.

The pipeline itself is in Overview; what each stage does is in Stages.

Topology #

                      ┌──────────────────────────┐
   bonsapi ──────────▶│        route queue       │
                      └──────────────────────────┘
                                   │
                             router worker
                                   │
                 ┌─────────────────┴─────────────────┐
                 ▼                                   ▼
            v1 arm queue                        v2 arm queue
      (unchanged — bonsai-invoice          (the Rust extraction
       and hinoki)                          worker)

Three queues. The route queue is new; the v1 arm keeps the queue and consumer that exist today, unchanged.

Producers. Everything that enqueues an extraction publishes to the route queue instead of directly to the v1 queue: a new extraction, a re-extraction, a document or invoice update that re-triggers extraction, and the children of an auto-split. Repointing the shared enqueue helpers rather than each call site is what keeps that list from drifting, and the job payload is unchanged — the router forwards it verbatim to whichever arm it picks.

The router is in front of 100% of extractions. That is the single most important fact about it, and every decision below follows from it: it must be fast, it must be stateless apart from one read, and it must fail to the v1 arm.

Arm assignment #

The router names its arms current and new, not v1 and v2. It outlives this experiment: the next pipeline change wants the same worker, the same weights and the same kill switch, and an arm named for a specific version has to be renamed or reinterpreted every time. current is whatever production runs today; new is whatever is being trialled against it.

Traffic splits by weights in basis points — 9,000 / 1,000 sends 90% to the pipeline running today and 10% to the one being trialled. Weights rather than a single rollout percentage because a third arm is then a third entry and a cumulative walk over the table, not a second threshold bolted onto the first.

Weights that do not sum to 10,000 are a misconfiguration, and the router resolves a misconfiguration the way it resolves everything else — all traffic to current, loudly. A router that silently normalises a bad weight table is one that quietly runs an experiment nobody configured.

Assignment is decided in this order. The order is the design — each rule exists because the one below it would otherwise produce the wrong answer:

1. Kill switch on → current. Beats everything, stickiness included.
2. Entity on the deny list → current.
3. Extraction already carries a pipeline_version → the arm that produced it. Stickiness beats weights, so changing a weight never moves an extraction that has already run.
4. Not an AP bill → current. Only AP bill is in scope, and this is the guard that keeps it so.
5. Entity on the allow list → new.
6. Hash bucket falls in new’s share → new, else current.

The bucket comes from hashing the extraction id with a configured salt and taking it modulo 10,000. Deterministic, not random, for three reasons that all matter later:

• The arm for any extraction is re-derivable in analysis from its id and the salt, so a readout never has to trust that the router logged correctly.
• A retried message lands in the same arm. Under a random draw, a redelivery could flip arms mid-extraction and the two arms would write over each other.
• It is testable without mocking a clock or an RNG.

The salt exists so the experiment can be re-randomised deliberately. Changing it mid-experiment invalidates the comparison: already-assigned extractions keep their arm while new ones are drawn from a different assignment, and the two populations stop being comparable.

What gets persisted is not current / new #

current and new are positions in an experiment, not identities. Their meaning shifts the moment the next pipeline change starts: today’s new is tomorrow’s current, and a row recorded as new last quarter would then read as a claim about a different pipeline entirely.

So the arm names live in the router’s config, and the pipeline_version column records the concrete pipeline that actually ran — v1, v2 — which stays true however many experiments follow. One mapping from arm to pipeline, in one place, changed when an experiment ends. Everything downstream — stickiness, the readout, the accuracy views — reads the concrete value and never has to know which experiment produced it.

Why per extraction, not per entity #

This is the decision most likely to be quietly reversed by someone who reasonably thinks entity-level assignment is tidier. It is not, and the reason is statistical rather than aesthetic.

Entity-level assignment makes the entity the unit of randomisation. The variance you are fighting is then the variance between entities, and entities differ enormously in exactly the thing that drives extraction accuracy: their document mix. One entity’s suppliers send clean digital PDFs with one table; another’s send phone photos of thermal receipts. Assign by entity and a handful of high-volume, awkward-document entities landing in one arm shifts that arm’s accuracy by more than the 3% effect being measured.

Entity volumes are also heavily skewed. A small number of entities produce a large share of the 51,591 monthly extractions, so an entity-level split will not produce arms of comparable size without hand-balancing — and hand-balancing is what randomisation is for.

Per-extraction assignment makes document mix balanced by construction. Each arm gets a random sample of the same population, so the mixes match to within sampling error and the readout arrives far sooner.

What it costs, stated honestly: the same entity has some of its documents extracted by v1 and some by v2 during the experiment. That is acceptable because assignment is sticky per extraction — a user never sees one document change pipeline under them — and because the per-document behaviour difference is, when v2 works, an improvement. It would not be acceptable for a change with a visible UI difference; it is acceptable for one whose output shape is identical by design.

Stickiness #

pipeline_version is a new column on the extraction row, written by the router at first assignment and never rewritten by a later roll.

Three paths inherit rather than re-roll:

• Re-extraction enqueues a fresh job. Without inheritance the router rolls again, the user watches the pipeline change under them, and an extraction that appeared in both arms pollutes both.
• Multi-document split children. A split produces new extractions. Each child inherits its parent’s arm. This matters in both directions: v1 splits inside hinoki, v2 splits from prep through bonsapi, and both paths have to propagate the parent’s pipeline_version at creation.
• Message redelivery. Handled by determinism rather than by state — the same id and salt give the same bucket — but the row read still short-circuits it.

The kill switch deliberately breaks stickiness. During an incident everything goes to current, including extractions previously assigned to new. An extraction’s arm can therefore change across re-extractions during an incident, and any extraction whose arm changed is excluded from the comparison — the router records the pipeline that actually ran each job, and a mismatch against pipeline_version is the exclusion signal. Safety wins over readability; it just has to be visible that it happened.

Where the config lives #

The weights and the allow/deny lists are ordinary worker config from Doppler. That is fine for values that change on the scale of days.

The kill switch cannot be one of them, because a Doppler value takes a restart, and the point of a kill switch is that it works when a restart is the thing you cannot afford. It is read from Redis, cached briefly in process, with the Doppler value as the startup default and a failed read resolving to off means route to current. A kill switch that fails open is not a kill switch.

Failure policy #

Any error in the router routes to current. Not “most errors”:

The router sits in front of every extraction in the product, so its worst plausible outcome is “the experiment loses a data point”, never “an extraction does not happen”. A router bug is invisible to customers and visible only as an arm-balance anomaly — which is why arm counts are a monitored metric and not just a log line.

The router never marks an extraction failed, never writes status, and never retries a stage. It reads one row, picks a queue, publishes, acks.

Shadow fan-out #

Before any live A/B, a short shadow phase: the router publishes to both arms for a sampled share of extractions. v1 writes normally; v2 runs the same worker with a shadow flag and writes to a shadow table instead of through bonsapi.

The shadow record carries the same extracted data, confidences and bbox the live path would have written, plus per-stage cost and timings — everything the comparison needs, and nothing the customer’s extraction can see.

This is the only point at which both pipelines run on the same document. In the live A/B they never do — each extraction goes to exactly one arm — so every comparison there is between-groups and subject to sampling error. The shadow slice gives paired comparison on identical input, which is what makes it the right and only place to answer:

• Does v2 produce the same shape v1 does — same fields populated, same confidence structure, same bbox coverage? This is the write-path parity check, and it is the biggest risk in the whole design.
• Where the two disagree on a field, which one does the verified data agree with?
• Do the prep split decisions match? The prep parity numbers come from here.
• What does v2 actually cost per page, measured rather than budgeted?

Shadow runs are pure cost with no customer benefit, so the sample stays small and the phase short. It ends when parity is demonstrated, not on a date.

A shadow run must never touch the customer’s extraction. It writes to the shadow table and nowhere else — no status updates, no heartbeats, no knowledge writes. The stage rules already forbid writes outside the write-back step, which is what makes a shadow run safe to add rather than a second write path to audit.

What invalidates the readout #

Worth stating up front, because each of these is cheap to prevent and impossible to fix after the fact:

The last one is the easy mistake: v1 has months of history and v2 has days. Every comparison is against v1 in the same window, never against the historical baseline. The 93.1% figure is the target; it is not the control group.