Routing & A/B Assignment #

Classification: Internal-Only

How an extraction gets assigned to an arm, why the assignment is per extraction rather than per entity, and what makes the resulting comparison readable.

The pipeline itself is in Overview; what each stage does is in Stages.

Topology #

                      ┌──────────────────────────┐
   bonsapi ──────────▶│        route queue       │
                      └──────────────────────────┘
                                   │
                             router worker
                                   │
                 ┌─────────────────┴─────────────────┐
                 ▼                                   ▼
            v1 arm queue                        v3 arm queue
      (unchanged — bonsai-invoice          (the Rust extraction
       and hinoki)                          worker)

Three queues. The route queue is new; the v1 arm keeps the queue and consumer that exist today, unchanged.

Producers. Everything that enqueues an extraction publishes to the route queue instead of directly to the v1 queue: a new extraction, a re-extraction, a document or invoice update that re-triggers extraction, and the children of an auto-split. Repointing the shared enqueue helpers rather than each call site is what keeps that list from drifting, and the job payload is unchanged — the router forwards it verbatim to whichever arm it picks. An OCR job, below, is rebuilt for the OCR queue instead.

The route queue carries that payload inside an envelope: the few fields the decision reads — extraction id, entity, extraction type, and the pipeline already recorded for the extraction — beside the payload, which the router opens only to look for pages: an OCR job, published on current and dropped on new. Four job shapes reach extraction and the router sits in front of all of them, so naming that union in the router would mean changing the router whenever any producer adds a field. Carrying the routing fields on the outside is also what lets the decision read nothing: without them the router would have to rebuild what the producer already had.

The router is in front of 100% of extractions. That is the single most important fact about it, and every decision below follows from it: it must be fast, it must be stateless apart from one read, and it must fail to the v1 arm.

Arm assignment #

The router names its arms current and new, not v1 and v3. It outlives this experiment: the next pipeline change wants the same worker, the same weights and the same kill switch, and an arm named for a specific version has to be renamed or reinterpreted every time. current is whatever production runs today; new is whatever is being trialled against it.

Traffic splits by weights in basis points — 9,000 / 1,000 sends 90% to the pipeline running today and 10% to the one being trialled. Weights rather than a single rollout percentage because a third arm is then a third entry and a cumulative walk over the table, not a second threshold bolted onto the first.

Weights that do not sum to 10,000 are a misconfiguration, and the router resolves a misconfiguration the way it resolves everything else — all traffic to current, loudly. A router that silently normalises a bad weight table is one that quietly runs an experiment nobody configured.

Assignment is decided in this order. The order is the design — each rule exists because the one below it would otherwise produce the wrong answer:

1. Kill switch on → current. Beats everything, stickiness included.
2. Entity on the deny list → current.
3. Not an AP bill → current. Only AP bill is in scope, and this is the guard that keeps it so. It sits above stickiness rather than below it: a job’s extraction type is read from the extraction when it is enqueued, while pipeline_version is written once and never rewritten, so the two disagree the moment anything changes an extraction’s type — switching type is a supported operation. Inheriting new for a type the v3 arm cannot extract dead-letters the job permanently, and an extraction that does not happen is worse than an experiment that loses a data point.
4. No extraction id on the job → current. A document reprocess is enqueued before any extraction exists, so there is nothing to hash and no extraction row for the readout to re-derive an arm from. It sits above stickiness for the same reason as rule 3.
5. Extraction already carries a pipeline_version → the arm that produced it, which the producer sends as assigned (see Stickiness). Stickiness beats weights, so changing a weight never moves an extraction that has already run.
6. Entity on the allow list → new.
7. Hash bucket falls in new’s share → new, else current.

The bucket comes from hashing the extraction id with a configured salt and taking it modulo 10,000. Deterministic, not random, for three reasons that all matter later:

• The arm for any extraction is re-derivable in analysis from its id and the salt, so a readout never has to trust that the router logged correctly.
• A retried message lands in the same arm. Under a random draw, a redelivery could flip arms mid-extraction and the two arms would write over each other.
• It is testable without mocking a clock or an RNG.

The salt exists so the experiment can be re-randomised deliberately. Changing it mid-experiment invalidates the comparison: already-assigned extractions keep their arm while new ones are drawn from a different assignment, and the two populations stop being comparable.

The bucket is a contract #

The readout re-derives arms from this formula, so it is a contract with the analysis code, not an implementation detail. Changing any part of it (the separator, the digest, the byte count, the endianness) silently invalidates every comparison in flight:

bucket = u64_be(sha256(salt || ":" || extraction_id)[0..8]) % 10_000
arm    = new if bucket < new_bp else current      # rule 7 only

extraction_id is the lowercase hyphenated UUID. A reference implementation for the readout side:

import hashlib

def bucket(extraction_id: str, salt: str) -> int:
    digest = hashlib.sha256(f"{salt}:{extraction_id}".encode()).digest()
    return int.from_bytes(digest[:8], "big") % 10_000

The router’s tests pin the same vectors, so a change on either side fails loudly.

What gets persisted is not current / new #

current and new are positions in an experiment, not identities. Their meaning shifts the moment the next pipeline change starts: today’s new is tomorrow’s current, and a row recorded as new last quarter would then read as a claim about a different pipeline entirely.

So the arm names live in the router’s config, and the recorded pipeline_version records the concrete pipeline that actually ran — v1, v3 — which stays true however many experiments follow. One mapping from arm to pipeline, in one place, changed when an experiment ends. Everything downstream — stickiness, the readout, the accuracy views — reads the concrete value and never has to know which experiment produced it.

Why per extraction, not per entity #

This is the decision most likely to be quietly reversed by someone who reasonably thinks entity-level assignment is tidier. It is not, and the reason is statistical rather than aesthetic.

Entity-level assignment makes the entity the unit of randomisation. The variance you are fighting is then the variance between entities, and entities differ enormously in exactly the thing that drives extraction accuracy: their document mix. One entity’s suppliers send clean digital PDFs with one table; another’s send phone photos of thermal receipts. Assign by entity and a handful of high-volume, awkward-document entities landing in one arm shifts that arm’s accuracy by more than the 3% effect being measured.

Entity volumes are also heavily skewed. A small number of entities produce a large share of the 51,591 monthly extractions, so an entity-level split will not produce arms of comparable size without hand-balancing — and hand-balancing is what randomisation is for.

Per-extraction assignment makes document mix balanced by construction. Each arm gets a random sample of the same population, so the mixes match to within sampling error and the readout arrives far sooner.

What it costs, stated honestly: the same entity has some of its documents extracted by v1 and some by v3 during the experiment. That is acceptable because assignment is sticky per extraction — a user never sees one document change pipeline under them — and because the per-document behaviour difference is, when v3 works, an improvement. It would not be acceptable for a change with a visible UI difference; it is acceptable for one whose output shape is identical by design.

Stickiness #

pipeline_version is not a column. Every AI snapshot records the pipeline that ran it in its private metadata, written by the worker that produced it, and the first AI snapshot of an extraction is its recorded arm. The producer (bonsapi) reads it when it enqueues and sends it on the job as assigned, so the router stays free of database access:

• No AI snapshot yet → unassigned, so the hash rolls. Deterministic, so a retry or a re-extraction clicked while the first run is in flight lands in the same arm unless the weights or allow list changed in between.
• A snapshot without the key → v1. Extractions that ran before the key was written need no backfill.
• The read fails → v1, logged. A lookup error never promotes an extraction to the new pipeline.

Two paths inherit rather than re-roll:

• Re-extraction enqueues a fresh job. Without inheritance the router rolls again, the user watches the pipeline change under them, and an extraction that appeared in both arms pollutes both.
• Message redelivery. Handled by determinism rather than by state — the same id and salt give the same bucket — and by the recorded arm once a snapshot exists.

The validation escalation needs nothing extra: it re-enqueues its own stamped job straight to the v3 queue, so it keeps its arm by construction.

Multi-document split children are independent extractions. A split deletes the original and creates new extractions, each of which rolls its own arm. A split parent writes no snapshot, so there is no recorded arm to inherit.

The kill switch deliberately breaks stickiness. During an incident everything goes to current, including extractions previously assigned to new. An extraction’s arm can therefore change across re-extractions during an incident, and any extraction that did not run on the arm it was assigned is excluded from the comparison. Each AI snapshot records the pipeline that actually ran it, so there are two signals: the snapshots disagree on pipeline_version (the arm changed later), or the first snapshot differs from the arm the hash gives the extraction id (the first run was forced to current by the kill switch, the deny list or a failed read of the recorded arm, and stickiness then kept it there). Allow-listed and denied entities are excluded by entity. Safety wins over readability; it just has to be visible that it happened.

Where the config lives #

The weights, the allow/deny lists, the salt and the kill switch are all ordinary worker config from Doppler, read once at startup. A config that does not parse — a missing salt, a weight table that does not sum to 10,000, an entity id with a typo in it — produces no config at all, and the worker resolves that the way it resolves everything else: kill switch on, everything to current, logged at error.

Flipping the kill switch therefore takes a restart, after the secret has synced (see Switches and weights). That is a real cost and it was accepted deliberately: at the volumes here — roughly 1.2 extractions a minute in total, a tenth of them in the new arm at 1,000 basis points — a rollout-length flip exposes well under one extraction, and every error path already lands on current without anyone touching the switch. A live-read source (Redis, a flag service) becomes worth its dependency when the new arm’s weight grows or when the arm does something a customer sees rather than something measured afterwards; until then it is a moving part protecting a tenth of an extraction a minute.

Switches and weights #

Everything the router reads, all prefixed EXTRACTION_ROUTER_:

A basis point is a hundredth of a percent: 10,000 is everything, 100 is 1%. The weights have no defaults and the shadow keys do. A missing weight would start an experiment nobody configured, whereas a missing shadow key means no shadow, which is the safe reading. Defaults here are the router’s own; the manifest requires both switch keys, so a missing one fails the pod before the router reads its config.

The two shares are independent and are not on the same scale:

• CURRENT_BP and NEW_BP split all eligible traffic, so NEW_BP = 1000 sends a tenth of it to new.
• SHADOW_BP is a share of the jobs that landed on current. At CURRENT_BP = 10000 that is the whole eligible stream, so SHADOW_BP = 100 shadows 1% of AP bills. At 9,000 / 1,000 it is 1% of the 90% on current, so 0.9% of the whole.

The two switches combine as follows:

Shadow and A/B are meant to be exclusive: a job runs on new as the live run, or it is shadowed from current, never both. A job on the new arm is delivered live to the v3 queue, stamped V3, and is never also shadowed.

Shadow mode still applies the A/B weights. With A off and B off, a non-zero NEW_BP assigns some jobs to new as live runs as well as shadowing from current. Keep CURRENT_BP = 10000 and NEW_BP = 0 in shadow mode.

A job on the new arm is a live run: the worker honours the router’s shadow flag, so it writes through bonsapi. The router’s kill switch is what keeps live runs off.

Both switches come from bonsai-secret and are required: a pod whose secret lacks either key does not start, so neither can silently read as off. The secret syncs from Doppler every hour (the ExternalSecret sets no refreshInterval) and on deploy, and KEDA/VPA start new pods at any time, so a Doppler change lands within an hour and reaches new pods only; running pods keep the old value until they restart. To apply it now, run tools/app-infra/eks-resources/refresh_secrets.sh, which only requests a sync and prints Done even if Doppler fails. Check that bonsai-secret holds the new value, then restart every router pod, or arms mix, and check each pod’s startup log: router config loaded shows the effective switches and weights, and router config could not be read means it failed closed. The weights and salt are set in Doppler. Shadow mode takes A and B both off, CURRENT_BP = 10000, NEW_BP = 0, the salt, and a small SHADOW_BP. A weight or salt that is missing or wrong makes the whole config fail closed, and that disables shadow too. The reverse does not hold: a bad shadow value disables only shadow.

Failure policy #

Any error in the router routes to current. Not “most errors”:

The router sits in front of every extraction in the product, so its worst plausible outcome is “the experiment loses a data point”, never “an extraction does not happen”. A router bug is invisible to customers and visible only as an arm-balance anomaly — which is why arm counts are a monitored metric and not just a log line.

The router never marks an extraction failed, never writes status, and never retries a stage. It reads one row, picks a queue, publishes, acks.

Shadow fan-out #

Before any live A/B, a short shadow phase: for a sampled share of extractions the router publishes to both arms. v1 runs and writes as it always does. v3 runs the same worker with a shadow flag, which writes nothing to production.

What it is for. The shadow phase answers one question: does v3 run without error on real documents? It catches worker-level breakage (a bad deploy, a missing secret, a queue misconfiguration) with no customer impact, which the A/B cannot. It does not measure accuracy: that is read from the A/B, and shadow keeps no result to compare. The v3 result is dropped after the run, and what remains is its log line, its spans and its metrics, all carrying the shadow flag. A shadow table, or a key in the page private metadata, is a later step if the logs prove too thin. The private metadata is also v1’s, and a shadow run must not touch the customer’s extraction.

This is the only point at which both pipelines run on the same document, so the shadow runs are also the place to read v3’s per-document latency and cost against v1’s, with the document mix held equal.

Shadow runs are pure cost with no customer benefit, so the share stays at 1–2% and the phase short. It ends when v3 is shown to run cleanly, not on a date.

Configuration #

The variables, their defaults, the units and the way the two switches combine are in Switches and weights above. In short: shadow runs only when both kill switches are off, and SHADOW_BP is a share of the jobs on the current arm. The A/B kill switch stops all v3 traffic, which a shadow run is, so it stops shadow too; the shadow switch stops only shadow. Flipping either takes a restart after the secret has synced, like the rest of the router’s config. A shadow value that does not parse (a share of 1% or above 10,000, a switch that is not true or false) shadows nothing and is logged at error level; it does not fail the config, so a typo there never stops the A/B.

Which jobs are shadowed #

A job is shadowed when all of these hold:

1. It was assigned to current. A job already running on v3 never runs on it twice.
2. It is an AP bill with an extraction id, as for the A/B.
3. Its entity is not on the deny list, so a denied entity’s documents never reach a second pipeline.
4. Its shadow bucket falls below the shadow share.

The shadow bucket is the A/B bucket under a salt of its own, a contract for the same reason:

shadow_bucket = u64_be(sha256(salt || ":shadow:" || extraction_id)[0..8]) % 10_000
shadowed      = shadow_bucket < shadow_bp

The separate domain keeps the shadow slice independent of the A/B slice. Sharing the A/B bucket would make the shadowed jobs exactly the ones the A/B sends to new first. The router’s tests pin the vectors:

The bucket hashes the extraction id, so every job of one extraction gets the same decision with no state to read: a re-extraction of a shadowed extraction is shadowed again.

Publishing #

The router publishes the live job to v1 first, with the retry it always had. Only once v1 has accepted it does it publish the shadow copy to the v3 queue, stamped shadow: true and pipeline_version: V3. A shadow publish failure never fails the message, because that would redeliver the job and publish the live v1 copy a second time. It is logged at warn level and the job is acknowledged.

The shadow copy keeps the payload’s job_id and trace_id. The two runs of one document share a trace id and a job id, which pair them in the logs.

What a shadow run does #

It runs every stage and writes nothing: no status update, no heartbeat, no page cache write, and none of the effects that follow a live write (usage, accounting publish, notifications, snippet jobs), which all happen behind bonsapi’s write path. The worker logs one line per attempt, Shadow run finished with an outcome or Shadow run failed, with the job id, trace id and invoice id. A shadow run that fails footing is counted as escalated and acknowledged, without a second attempt.

A shadow job that fails exhausts its retries like any other v3 job and lands in the dead-letter queue. That is the signal this phase exists to produce.

The worker runs each job in the mode the router stamped on it and has no shadow override of its own. A job without the flag is a live run that writes through bonsapi, so only the router’s kill switch and weights decide whether v3 writes.

Running the v3 arm locally #

The router and the worker read their settings from .env, which mise run dev rewrites from Doppler (bonsai, dev_local). To run AP bills live on v3, these three must read:

EXTRACTION_ROUTER_KILL_SWITCH=false
EXTRACTION_ROUTER_CURRENT_BP=0
EXTRACTION_ROUTER_NEW_BP=10000

Keep EXTRACTION_ROUTER_SHADOW_KILL_SWITCH at true so nothing is also shadowed. A container reads .env only when it is created, and the router keeps what it finds in the process environment over a later .env edit, so after a change run docker compose up -d --force-recreate tofu-extraction-router; a restart or a cargo watch rebuild is not enough. Check what the container actually has with docker exec tofu-extraction-router env | grep EXTRACTION_ROUTER. The worker also needs EXTEND_API_KEY and Bedrock credentials in .env. Only AP bills reach v3, and only those that have an extraction id. AR invoices, credit notes and documents that leave NEEDS_REVIEW stay on v1.

The router logs routed to the new arm for each job, and the worker logs a non-shadow outcome.

What invalidates the readout #

Worth stating up front, because each of these is cheap to prevent and impossible to fix after the fact:

The last one is the easy mistake: v1 has months of history and v3 has days. Every comparison is against v1 in the same window, never against the historical baseline. The 93.1% figure is the target; it is not the control group.