gRPC Read Contract

gRPC read contract: field selection for bonsapi #

Classification: Restricted (describes source code and access control).

INTERNAL - DO NOT FORWARD

Status: draft for review (ENG-9842, Phase 0 of ENG-9713). Every new external gRPC service in Phase 0 (0.3 to 0.7) follows this page. A change to a rule here is a change to this page first.

Why #

Callers cannot choose which fields a read returns, so the MCP strips them itself (withoutBboxes, capLineItems, capBytes, keepBboxes) and decides the reply shape in the wrong layer. With this contract the caller names the fields it wants, bonsapi loads and returns only those, and the MCP passes the caller’s choice straight through.

Scope: the new services for direct expenses, extractions, documents, bank statements and invoices (AP bill and AR invoice) under protobuf/external/. The existing entity, extraction status, knowledge and pricing services are not changed by this page. Webapps never call these services, and no REST route, REST response or OpenAPI entry changes (see the guardrail in ENG-9841).

1. Services and naming #

One service per resource, one package per service, versioned from the start:

Resource Proto file Package Service
Direct expense external/direct_expense/direct_expense.proto bonsapi_grpc.direct_expense.v1 DirectExpenseService
Extraction external/extraction/extraction.proto bonsapi_grpc.extraction.v1 ExtractionService
Document external/document/document.proto bonsapi_grpc.document.v1 DocumentService
Bank statement external/bank_statement/bank_statement.proto bonsapi_grpc.bank_statement.v1 BankStatementService
Invoice (AP bill, AR invoice) external/invoice/invoice.proto bonsapi_grpc.invoice.v1 InvoiceService

Method names follow the AIP standard methods so callers can guess them:

Method Shape
Get<Resource> Get<Resource>Request { string id; string entity_id; FieldMask read_mask; }
BatchGet<Resources> ids plus one read_mask, per-item results (§8)
List<Resources> request: entity_id, page_size, page_token, filter, order_by, read_mask. Reply: the resources, next_page_token, read_time, resume_time (§8)
List<Resource>Versions the append-only data history of one resource, newest first
Update<Resource>Data partial update with update_mask, etag, request_id, read_mask (§5)

Reserved names. Watch<Resources> (server streaming, for example rpc WatchExtractions(WatchExtractionsRequest) returns (stream WatchExtractionsResponse)) is reserved for the live status stream that tofu-web needs (ENG-9891). It is not built in Phase 0, and no other method may take a Watch prefix.

Resource shape. Each resource message is the parent record plus the data version a read selected, under data, with lines under data.line_items (data.transactions for bank statements). Field names match the REST response names so callers moving off REST keep their vocabulary. Bookkeeping metadata is a typed message, never a JSON string, so mask paths can reach into it (data.metadata.traces).

Explicit presence. Every scalar on a resource message is optional, so “not selected” (unset) is never confused with a real zero, empty string or false. A field that was selected but left out because the caller may not read it is listed in omitted_mask (§2), so an unset field that is not in omitted_mask really has no value: an unset link id means there is no link.

2. Read masks #

Every read request has a google.protobuf.FieldMask read_mask (AIP-157, partial responses).

Paths #

  • Paths are proto field names in snake_case, separated by dots, relative to the resource message. On a List or BatchGet, paths are relative to each returned resource, not to the response (data.total_amount, never direct_expenses.data.total_amount).
  • A path through a repeated message field selects that subfield on every element: data.line_items.description.
  • A path that ends at a message field selects every non-heavy field under it: data returns the whole data version except its heavy fields.
  • Map fields are selected whole. There are no map-key paths (data.metadata.traces yes, data.metadata.traces.title no).
  • * alone selects every non-heavy field of the resource. It cannot be combined with a prefix (data.* is invalid; write data).
  • Heavy fields (traces, bboxes, page coordinates, sync activity payloads) are returned only when their full path is named. Neither * nor a parent path expands to them.
  • Duplicate and overlapping paths are allowed and normalised (data, data.title is the same as data).

Always returned #

id, entity_id and etag are always set, whatever the mask, so a caller can always correlate and write back. On a selected data version, data.id and data.action_type are always set too.

Every resource also carries google.protobuf.FieldMask omitted_mask. It is set whenever a path selected through *, a parent path or the default mask was left out because the caller lacks its permission (§10), and it lists those paths. It is empty when nothing was left out.

Empty mask #

An omitted or empty mask means the default mask of that read type (§3). There is no way to ask for “nothing”.

Invalid masks #

Every bad mask fails the whole call with INVALID_ARGUMENT, a BadRequest.FieldViolation per bad path (read_mask.paths[2]), and the paths this caller may use in ErrorInfo.metadata["valid_paths"]. Nothing is silently ignored.

  • A path that does not exist on the resource, or a * misuse: reason INVALID_READ_MASK.
  • A path that exists but the caller may not read (§10): reason FIELD_NOT_ALLOWED.
  • A mask over the limits in §9: reason READ_MASK_TOO_LARGE.

valid_paths is computed for the caller, so it never lists a path they may not read.

Pushdown #

Each path maps to a load group, and bonsapi only runs the queries for the groups a mask touches:

Load group Paths What is not loaded when the group is not selected
header parent and data.* scalars always loaded (ids, status, etag)
metadata data.metadata.* the metadata jsonb is read with heavy keys removed in SQL (metadata - 'traces' - 'bboxes') unless they were named
lines data.line_items.* / data.transactions.* the line query is skipped
line_metadata data.line_items.metadata.* per-line metadata jsonb, same rule as metadata
sync data.sync_activities.* the sync activity query is skipped
creator data.creator.* the user lookup is skipped
pages pages.* (document, extraction) the page query is skipped

private_metadata is never part of any read type, so the new read queries never select it. It is the largest column on the measured tables (medians: 15.2 KB per bank statement data row, 10.1 KB per invoice data row, 17.9 KB per document page row), so not loading it is the main saving on the database side.

3. Default masks and sizes #

Defaults hold ids, status and the main bookkeeping fields: enough to review, match and publish a record, nothing about where a value was found. Heavy fields are never in a default.

Defaults per read type #

Direct expense (GetDirectExpense, ListDirectExpenses, BatchGetDirectExpenses)

id, entity_id, extraction_id, status, external_id, publish_sync_status, last_publish_at,
data.title, data.reference, data.transaction_date, data.currency_id, data.total_amount,
data.line_amount_type, data.spend_type, data.payment_type, data.contact_name,
data.external_contact_id, data.external_account_id, data.external_tax_rate_id,
data.line_items.id, data.line_items.description, data.line_items.quantity,
data.line_items.unit_price, data.line_items.amount, data.line_items.tax_amount,
data.line_items.external_account_id, data.line_items.external_tax_rate_id

Invoice (AP bill and AR invoice)

id, entity_id, extraction_id, type, status, external_id, publish_sync_status, last_publish_at,
data.title, data.invoice_number, data.invoice_reference, data.invoice_date, data.due_date,
data.currency_id, data.total_amount, data.line_amount_type, data.contact_name,
data.external_contact_id, data.external_tax_rate_id,
data.line_items.id, data.line_items.description, data.line_items.quantity,
data.line_items.unit_price, data.line_items.tax_amount, data.line_items.external_account_id,
data.line_items.external_tax_rate_id, data.line_items.external_item_id

Bank statement (statement header only; transactions are read with ListBankTransactions, which pages, so a 300-line statement never needs one huge reply). status is the extraction’s status, because the bank statement row has none.

id, entity_id, extraction_id, status, data.external_id,
data.bank_name, data.account_number, data.external_account_id, data.currency_id,
data.opening_balance, data.closing_balance, data.start_date, data.end_date, data.transaction_count

ListBankTransactions default: id, date, description, amount, reference, currency_id, external_contact_id, external_account_id, external_tax_rate_id.

Extraction

id, entity_id, type, status, title, error_phase, error_message, source, created_at, updated_at,
document_id, direct_expense_id, invoice_id, bank_statement_id

The link fields (document_id and the record ids) are what 1.2 needs to pair a document with its draft without guessing by creation time. Each record id needs the read permission of the record it points to (§10). A caller without it gets the extraction with that id unset and listed in omitted_mask.

Document

id, entity_id, file_name, mime_type, size, status, source, extraction_types, page_count,
extraction_ids, created_at, updated_at

Sizes #

Measured on stored rows: aggregate medians across many organisations, computed in SQL, with no records read out. The default column is the JSON size of exactly the default fields’ stored values. Derived fields (sync status, last publish time, link ids, counts) are not stored on these rows and add under 200 bytes. Medians of parts do not add up exactly, so the full-size totals are indicative.

Read type Sample Full stored size (median parts) Default mask (measured, median)
Direct expense, 2 lines 317 versions, 21 entities, last 14 days header 1.7 KB + metadata 2.0 KB (traces 1.6 KB, bboxes 0.4 KB) + lines 2.0 KB + line metadata 2.7 KB ≈ 8.4 KB, before REST adds the extraction, sync activities and creator (about 20,000 characters as the partner received it) 1.33 KB (p90 1.40 KB)
Invoice, 2 lines 347 versions, last day header 9.1 KB (mostly private_metadata) + traces 3.3 KB + bboxes 1.7 KB + lines 2.6 KB + line metadata 5.9 KB 1.27 KB (p90 1.38 KB)
Bank statement 1,935 versions, last 14 days, median 33 transactions header 16.1 KB (private_metadata 15.2 KB) + metadata 3.5 KB + 33 × (0.9 KB + 2.2 KB metadata) ≈ 122 KB header 0.39 KB; 0.29 KB per transaction (p90 0.34 KB), so a page of 50 is about 15 KB
Extraction 2,000 rows, last day 0.4 KB row 0.30 KB
Document 2,000 rows, last day, median 1 page 0.7 KB row + 15.1 KB per page (private_metadata 17.9 KB) 0.29 KB

The QA target is about 3,000 characters for a two-line receipt. The direct expense default measures 1.33 KB at the median and 1.40 KB at p90, under half the target. Proto JSON adds little to these sizes, and the binary wire format is smaller. 0.9’s reply size budget tests measure the real serialised replies and fail if a default grows past its budget.

Changing a default safely #

  • Adding a path to a default is allowed in any PR, as long as the size budget test still passes. Callers that want a fixed shape send an explicit mask.
  • Removing a path from a default is a deprecation. Announce it in the MCP changelog and the PR, keep it for at least 30 days, and update the golden snapshot in its own PR.
  • Every default lives in a golden snapshot (§11), so no default changes by accident.

4. How masks are applied (0.1.2) #

Decision: proto field options plus generated code. Not runtime reflection, not hand-written matching.

A new extension-only proto, protobuf/options/field_selection.proto, sits next to log_redaction.proto:

package bonsai.options.v1; // new files get no lint exemptions, so the package is versioned

message FieldSelection {
  bool read_default = 1;        // in this read type's default mask
  bool heavy = 2;               // only returned when its full path is named
  string load_group = 3;        // §2 pushdown group; inherited from the parent when empty
  string read_permission = 4;   // extra permission beyond the resource read, e.g. "MEMBER:READ"
  bool writable = 5;            // may appear in an update_mask
}

extend google.protobuf.FieldOptions {
  optional FieldSelection field_selection = 50001;
}

build-protobuf.rs already compiles every proto under protobuf/. It gains a step that asks tonic_prost_build for the file descriptor set, decodes it with prost-reflect as a build-dependency only (plain prost-types drops custom options, because prost discards unknown extension fields), and writes one module per resource into src/grpc/__generated/, committed like the rest of that directory:

  • PATHS: every valid path with its flags, load group and permission.
  • DEFAULT_MASK: the normalised default.
  • fn load_plan(&FieldMask) -> Result<LoadPlan, MaskError>: validates, expands * and parent paths, drops heavy fields unless named, returns the load groups to query.
  • fn prune(&mut Resource, &NormalisedMask): clears every field not in the mask, recursively, including inside repeated fields.

Why this option:

  • Paths are checked at build time against the real messages, so a renamed field breaks the build, not a caller.
  • No descriptor pool or dynamic messages at runtime, and no new runtime dependency.
  • It is the same pattern as log_safe: the rule lives next to the field, so the default mask, the permission and the redaction status of a field are read in one place.
  • prost-reflect at runtime was rejected: it would need a descriptor set loaded at start-up and DynamicMessage conversions on every reply, for no gain over generated code. Hand-written matching was rejected because it drifts from the schema.

The TS side reads the same option, the way scripts/generate-log-safe-fields.ts reads log_safe, so the MCP can describe valid fields values in tool schemas without a second list.

5. Writes #

Update<Resource>Data (0.7) is a partial update that writes a new version in the append-only data history:

message UpdateDirectExpenseDataRequest {
  string id = 1;
  string entity_id = 2;
  DirectExpenseData data = 3;                // only the update_mask paths are read
  google.protobuf.FieldMask update_mask = 4; // required, explicit paths only, no "*"
  string etag = 5;                           // required, from the last read (§7)
  string request_id = 6;                     // required UUID (§7)
  google.protobuf.FieldMask read_mask = 7;   // shape of the reply; empty = default mask
}
  • update_mask is required and lists explicit paths. Only fields marked writable are allowed, otherwise INVALID_ARGUMENT, reason INVALID_UPDATE_MASK.
  • A field in update_mask but unset in data is cleared. A field set in data but not in update_mask is ignored.
  • data.line_items in the mask replaces the whole line list. A line that carries an existing id keeps its stored metadata (bboxes and traces). New lines have no id.
  • data.line_items.<field> updates that field only on the lines sent, matched by id. Lines not sent are unchanged, and an unknown line id is INVALID_ARGUMENT.
  • data.line_items and any data.line_items.<field> in the same update_mask contradict each other (replace the list or patch it), so that mask is INVALID_UPDATE_MASK. The same rule applies to any message path listed with one of its own subpaths. Duplicate identical paths are allowed.
  • Heavy fields are never writable. The server carries bboxes and traces forward for every field the update did not change, which moves the MCP’s keepBboxes logic into bonsapi.
  • The reply is the updated resource with read_mask applied.
  • The gRPC write calls new service functions. The REST write path and the is_extraction_updating field the webapp waits on keep their behaviour.

6. Errors (0.1.5) #

Every error is a google.rpc.Status: a code, a message for humans, and typed details (bonsapi uses tonic-types to attach them):

  • google.rpc.ErrorInfo, always present: reason (from the table below), domain = bonsapi.gotofu.com, metadata with string keys only.
  • google.rpc.BadRequest with FieldViolations for every INVALID_ARGUMENT caused by a request field.
  • google.rpc.RetryInfo on every retryable error.
  • google.rpc.RequestInfo: the request id, so support can find the trace.

Stable reasons. A reason is an API. It is never renamed or reused, and callers branch on reason, never on the message.

Code Reason When Metadata
INVALID_ARGUMENT INVALID_READ_MASK unknown path or * misuse invalid_paths, valid_paths
INVALID_ARGUMENT FIELD_NOT_ALLOWED a path that was named explicitly needs a permission the caller lacks invalid_paths, valid_paths (only paths this caller may read)
INVALID_ARGUMENT INVALID_UPDATE_MASK unknown, non-writable, missing, or overlapping path invalid_paths, writable_paths
INVALID_ARGUMENT READ_MASK_TOO_LARGE over the path count or depth limit max_paths, max_depth
INVALID_ARGUMENT INVALID_PAGE_TOKEN malformed, expired, or issued for another filter or order
INVALID_ARGUMENT INVALID_FILTER / INVALID_ORDER_BY syntax error or a field that is not allow-listed allowed_fields
INVALID_ARGUMENT BATCH_TOO_LARGE more ids than the batch limit max_ids
INVALID_ARGUMENT RESPONSE_TOO_LARGE one resource alone exceeds the size limit (§9). On BatchGet this is a per-item error size_bytes, limit_bytes
INVALID_ARGUMENT REQUEST_ID_REUSED same request_id, different request
INVALID_ARGUMENT ETAG_REQUIRED a write without etag
UNAUTHENTICATED UNAUTHENTICATED missing or invalid token
PERMISSION_DENIED ENTITY_ACCESS_DENIED no read on this entity or resource type resource_type
NOT_FOUND RESOURCE_NOT_FOUND no such id in this entity (also used when the resource is in another entity, so ids cannot be probed) resource_type
ABORTED ETAG_MISMATCH the resource changed since the caller’s read current_etag, plus a ConflictDetail (§7)
ABORTED REQUEST_IN_PROGRESS a duplicate of a write that is still running RetryInfo
INVALID_ARGUMENT QUOTA_COST_TOO_HIGH one call costs more units than the whole bucket holds, so waiting cannot help cost, capacity
RESOURCE_EXHAUSTED QUOTA_EXCEEDED over the organisation’s quota quota, RetryInfo
DEADLINE_EXCEEDED DEADLINE_EXCEEDED the server deadline passed
UNAVAILABLE UPSTREAM_UNAVAILABLE database, cache or accounting provider not reachable RetryInfo
INTERNAL INTERNAL anything else request_id only

Per-item results. Batch calls never fail as a whole because one item failed. Each item has its own result in request order, oneof { Resource resource; google.rpc.Status error; }. Only request-level problems (bad mask, too many ids, quota) fail the call.

Nothing internal leaves the service. No stack traces, SQL, table or column names, internal service names, file paths or upstream error bodies go into a message or into metadata. INTERNAL carries only the request id; the details stay in the logs, joined by trace id. The existing map_api_error (grpc/auth.rs) currently puts err.to_string() into INTERNAL. The new services use a new mapper that follows this table, and map_api_error stays as it is for the existing services.

7. Concurrency and idempotency (0.1.6) #

Etags. Every resource returns an opaque etag. It is a hash of everything a write or a status change can alter, not of a timestamp:

  • the id of the latest data version (every data edit appends a new version);
  • every column of the parent row except created_at and updated_at, so a parent column added later is covered without a spec change;
  • the linked extraction’s status, which changes on mark ready, done and draft without a new data version;
  • the id of the latest sync activity, which changes on publish and import without a new data version.

What that covers today, per type:

Type Parent columns in the hash Also hashed
Direct expense extraction_id, external_id, source, is_ai_autosplit, status latest data id, extraction status, latest sync activity id
Invoice extraction_id, external_id, status, source, is_ai_autosplit, last_publish_at, last_import_at, publish_error_type, import_error_type, publish_sync_status, import_sync_status latest data id, extraction status, latest sync activity id
Bank statement extraction_id, is_ai_autosplit, metadata, version (no status or external_id on this row: status is the extraction’s, and external_id is on the data version) latest data id, extraction status, latest sync activity id
Extraction every extraction column none
Document every document column none

The stored updated_at column is not used, because nothing guarantees every status or sync change bumps it.

changed_at. For the same reason, the API’s updated_at (returned, filtered and sorted on) is backed by a new changed_at column on each parent table, not by the stored updated_at. Database triggers bump it in the same transaction as any change to an etag input: the parent row, a new data version, the extraction’s status, or a new sync activity. So whenever the etag changes, updated_at moves forward too, and a publish or mark-done shows up in a “changes since” read. It is a new column, so the REST responses and the webapp keep reading the old one unchanged. 0.2 adds the column, its backfill and its index. Callers treat the etag as an opaque string. It is computed in the same query that loads the header group, so it costs no extra query.

Preconditions. Every write takes the etag the caller last read and is required on updates (ETAG_REQUIRED without it). The etag is recomputed inside the write transaction, after SELECT … FOR UPDATE on the parent row, so two writers cannot both pass the check. If it no longer matches, the write is refused with ABORTED / ETAG_MISMATCH, and the details carry a ConflictDetail. The caller can then merge and retry without another read.

Each service package defines its own ConflictDetail, so the current resource is typed:

// bonsapi_grpc.direct_expense.v1, and the same per service.
message ConflictDetail {
  string current_etag = 1;
  // The resource as it is now, with the request's read_mask and the caller's
  // field permissions applied (§10).
  DirectExpense current = 2;
}

Request ids. Every write takes a client-generated UUID request_id. bonsapi stores (organization_id, user_id, method, request_id) → hash of the request and the outcome, for 24 hours. The user id is in the key so two users of one organisation can never collide on an id.

  • Stored outcomes are only final ones: a success, or a refusal that the same request would get again (INVALID_ARGUMENT, PERMISSION_DENIED, NOT_FOUND, ABORTED / ETAG_MISMATCH).
  • Transient failures are not stored (UNAVAILABLE, DEADLINE_EXCEEDED, INTERNAL, RESOURCE_EXHAUSTED). The write’s own transaction has rolled back and the claim is released, or its lease lapses (below), so a retry with the same id runs the write again. This is the timeout case request ids exist for.
  • Same id and same request, with a stored outcome: that outcome is returned and nothing is applied a second time.
  • Same id and a different request: INVALID_ARGUMENT / REQUEST_ID_REUSED.
  • Same id while the first call still holds a live claim: ABORTED / REQUEST_IN_PROGRESS, with RetryInfo set to when the lease ends.

Claim and lease. The claim and the outcome are written at different times:

  1. The claim is committed in its own short transaction before the write starts, so a concurrent duplicate can see it. It carries a lease that ends at the method’s server deadline (§9) plus 5 seconds.
  2. The outcome is written inside the write’s own transaction, so a write is never applied without its outcome, and an outcome is never stored for a write that rolled back.
  3. If the process dies mid-write, the claim has no outcome and its lease lapses within seconds, not 24 hours. The next retry takes over the claim and runs the write, because the rolled-back write applied nothing.
  4. A retry first looks for a stored outcome, then for a live lease. So a write that committed just before a crash is answered from its outcome and never applied twice.

Reads are naturally idempotent and take no request id.

8. Pagination, batch get, filter and sort (0.1.7) #

Keyset pagination. page_size defaults to 20 and is capped at 100. A larger value is lowered to 100; this is standard AIP-158 behaviour and is not a silent cut, because next_page_token says there is more. page_token is opaque: a signed, base64url token holding the last row’s sort key and id, the read_time (below), and a hash of filter and order_by. Every order ends with id as a tiebreak. An empty next_page_token means the last page.

Both sort keys are stable within one paging run:

  • created_at never changes. Rows created after the first page are excluded by read_time, so pages never repeat a row.
  • updated_at (backed by changed_at, §7) changes whenever the etag changes. The first page records read_time (the server time when that page is read), and every later page adds updated_at <= read_time. A row edited during the run therefore moves above read_time and leaves the result. It never appears twice, and the order of the rest never shifts.

Late commits. Both timestamps are stamped when a write’s transaction runs, not when it commits. A row stamped just below read_time whose transaction commits after the cursor has passed its position is missed by the run. So every reply also returns resume_time = read_time minus a 5-minute commit-lag margin:

  • After the last page, a caller that needs everything lists again with filter: "updated_at > <resume_time>" (or created_at). That is the incremental “changes since” read tofu-web needs.
  • The overlap window returns some rows a second time, so callers dedupe by id and etag. The same pair is the same version and is skipped.
  • The margin must be longer than any transaction that stamps these columns. 0.2 checks that the database’s statement and idle-in-transaction timeouts keep every such transaction below it, and raises the margin if not.

A token used with a different filter or order is INVALID_PAGE_TOKEN. There is no total_size, because counts on large entities are expensive.

Batch get. BatchGet<Resources>Request { string entity_id; repeated string ids; FieldMask read_mask; } takes 1 to 100 distinct ids, one mask for all of them, and returns per-item results (§6) in request order. If the results together pass the size limit, items are filled in request order. The first item that no longer fits, and every item after it, gets a per-item RESPONSE_TOO_LARGE error, so the caller re-requests exactly those ids. A duplicated id fails the call with INVALID_ARGUMENT.

Filter. An AIP-160 subset: field op value comparisons joined by AND, with OR only between values of the same field. Operators: =, !=, <, <=, >, >=. Values are quoted strings, numbers, booleans or enum names. Only allow-listed fields can be filtered, per list method, and each one is backed by an index. Field permissions (§10) apply to filter and order_by as they do to masks: naming a field the caller may not read (for example extraction_id without INVOICE:READ) fails with INVALID_ARGUMENT / FIELD_NOT_ALLOWED, so a filter can never probe a hidden value. The allowed_fields in INVALID_FILTER errors are computed for the caller in the same way.

List Filter fields
ListDirectExpenses, ListInvoices, ListBankStatements status, created_at, updated_at, extraction_id, publish_sync_status, data.action_type, data.currency_id (plus type for invoices)
ListExtractions type, status, created_at, updated_at, document_id
ListDocuments status, source, created_at, updated_at

Example: status = "READY" OR status = "DONE" AND created_at >= 1794787200000. OR binds tighter than AND here, as in AIP-160.

Reply size on lists. A List returns as many rows as fit in the size limit (§9), in order. If the page fills before page_size rows, it ends early with a next_page_token, so nothing is skipped and the reply says there is more. Only a single row that is over the limit on its own fails the call, with RESPONSE_TOO_LARGE and that row’s id in the metadata.

Sort. order_by is one allow-listed field with an optional desc: created_at (default created_at desc) or updated_at. id is always added as the last key. Only one sort field is allowed, because the read_time rule above relies on a single key.

9. Limits and resilience (0.1.8) #

Limit Value Over the limit
Server deadline: Get, BatchGet 5 s, 10 s DEADLINE_EXCEEDED
Server deadline: List, List…Versions 10 s DEADLINE_EXCEEDED
Server deadline: Update… 15 s DEADLINE_EXCEEDED
Page size default 20, max 100 lowered to 100
Batch ids 100 BATCH_TOO_LARGE
Mask paths 64 READ_MASK_TOO_LARGE
Mask depth 5 segments READ_MASK_TOO_LARGE
Path length 256 characters INVALID_READ_MASK
Reply size 1 MiB, or less if the caller sends tofu-max-response-bytes metadata Get: RESPONSE_TOO_LARGE. List: a shorter page with a next_page_token (§8). BatchGet: per-item RESPONSE_TOO_LARGE on the items that did not fit (§8). Never a cut resource.

The server uses the shorter of the caller’s grpc-timeout and its own deadline. The reply size is checked after pruning and before sending. Errors name the size and the limit, so the caller can narrow the mask or the page size. A tofu-max-response-bytes value above 1 MiB is lowered to 1 MiB. The MCP sends its own budget (today DEFAULT_MAX_RESPONSE_BYTES, 200,000) in that metadata instead of cutting the reply itself.

Retries. Callers retry only:

  • reads, on UNAVAILABLE, DEADLINE_EXCEEDED and ABORTED / REQUEST_IN_PROGRESS;
  • writes that carry a request_id, on the same codes;
  • any call that returns RESOURCE_EXHAUSTED with RetryInfo, after the delay it gives.

Backoff is exponential with full jitter: base 100 ms, factor 2, cap 2 s, at most 3 attempts, always within the caller’s own deadline. INVALID_ARGUMENT, PERMISSION_DENIED, NOT_FOUND and ABORTED / ETAG_MISMATCH are never retried as they are.

Per-organisation quotas. A Redis token bucket per organisation, per service: 600 read units and 120 write units per minute. Costs:

Call Bucket Units
Get, List…Versions page read 1
BatchGet read 1 per id
List read 1 per 20 rows of the requested page_size, rounded up (a page of 100 costs 5)
Update… write 1

Units are charged before the call runs. A call that costs more than the whole bucket holds is INVALID_ARGUMENT / QUOTA_COST_TOO_HIGH, never a RESOURCE_EXHAUSTED that no wait could clear. This sits on top of the MCP’s per-user limit (see Rate Limiting), so one organisation with many users cannot starve the others. Over quota is RESOURCE_EXHAUSTED / QUOTA_EXCEEDED with RetryInfo set to when the bucket refills. Like the MCP limiter, it fails open if Redis is down, and the gap is logged and counted. The numbers are starting values, to be set from the 0.9 metrics before tofu-web uses these services.

10. Field-level authorization and privacy (0.1.9) #

Resource permissions. Every call first checks the same entity-scoped resource read that REST uses for that resource (grpc/auth.rs → AccessControl):

Service Read Write
DirectExpenseService DIRECT_EXPENSE:READ DIRECT_EXPENSE:UPDATE
InvoiceService INVOICE:READ INVOICE:UPDATE
BankStatementService BANK_STATEMENT:READ BANK_STATEMENT:UPDATE
ExtractionService INVOICE:READ (as REST extractions) n/a
DocumentService DOCUMENT:READ n/a

Fields that need more than the resource read.

Paths Extra permission Why
data.creator.*, data.sync_activities.creator.* MEMBER:READ names another user of the organisation
data.sync_activities.* INTEGRATION:READ shows integration activity and provider error types
pages.* on extraction DOCUMENT:READ page geometry and file references are document data
extraction_id on direct expense, invoice and bank statement INVOICE:READ links to the extraction, which REST gates on INVOICE
direct_expense_id on extraction DIRECT_EXPENSE:READ links to a record the caller may not read
invoice_id on extraction INVOICE:READ same; always held by an extraction reader today, but stated so the rule does not depend on that
bank_statement_id on extraction BANK_STATEMENT:READ same
extraction_ids on document INVOICE:READ same, as REST maps extractions to INVOICE

publish_sync_status and last_publish_at need only the resource read. They summarise the record’s own publish state, as REST already returns with that read, and carry no provider detail. That detail lives in data.sync_activities.*, which is gated above.

Never exposed on any path: private_metadata, user email addresses, storage file paths, signed URLs (these keep their own RPC), and provider credentials or raw provider responses.

A mask can never widen access. Paths are resolved against the caller’s permissions after the mask is validated:

  • A forbidden path that was named explicitly fails the call with INVALID_ARGUMENT / FIELD_NOT_ALLOWED, which lists the forbidden paths and the paths this caller may read. This is the same error shape as an unknown path, as the ticket specifies, so a caller has one way to correct a mask.
  • A forbidden path reached through *, a parent path or the default mask is left out and listed in omitted_mask, so a default read never fails because of a field the caller did not ask for, and the caller can still tell “not permitted” from “no value”.
  • Pruning runs on the final reply, after the load plan, so a field loaded for another reason (for example to compute the etag) cannot leak.
  • On batch and list calls, a resource in an entity the caller cannot read is a per-item NOT_FOUND (batch) or is never matched (list).

Personal-data allow-list. These fields can hold personal data and are never log_safe: title, description, reference, contact_name, bill_address, ship_address, ship_from_address, tax_identification_number, learning, invoice_number, invoice_reference, account_number, bank_name, transaction description, contact_name, reference and check_number, metadata.traces (raw text read from the document), metadata.invoice_lines_aggregated_description, file_name, error_message, creator.*. A new field is personal data unless the PR argues otherwise. The 0.9 field catalog marks each field.

Logging and log_safe. Masks do not change redaction:

  • Redaction stays per field. A field is logged verbatim only if it carries (bonsai.options.log_safe) = true. A masked reply is a subset of the full message, so the same allow-list covers it.
  • read_mask and update_mask paths are field names, not values, so they are log_safe and are logged. This is what makes the mask-usage metrics in §12 possible.
  • bonsapi does not log reply payloads, only sizes, path counts and error reasons. The MCP’s redact.ts keeps applying the allow-list generated from the protos (scripts/generate-log-safe-fields.ts).
  • Being in a default mask does not make a field log-safe. read_default and log_safe are independent options.

Test plan with a restricted user (implemented in 0.3 to 0.6, one fixture per service):

  1. A user with a role that has the entity but not the resource read gets PERMISSION_DENIED / ENTITY_ACCESS_DENIED, on Get, BatchGet and List.
  2. A user with resource read but no MEMBER:READ:
    • a default read succeeds, with data.creator unset and listed in omitted_mask;
    • read_mask: "*" succeeds, with data.creator unset and listed in omitted_mask;
    • read_mask: "data.creator.name" fails with INVALID_ARGUMENT / FIELD_NOT_ALLOWED, invalid_paths = data.creator.name, and valid_paths does not contain any data.creator path.
  3. A user with extraction read but no DIRECT_EXPENSE:READ: a default GetExtraction succeeds with direct_expense_id unset and listed in omitted_mask, and read_mask: "direct_expense_id" fails with FIELD_NOT_ALLOWED.
  4. A user with access to entity A only: BatchGet with ids from A and B returns resources for A and per-item NOT_FOUND for B. List on B fails with ENTITY_ACCESS_DENIED.
  5. A path to private_metadata, or to any field that is not in the schema, is INVALID_READ_MASK, and its name does not appear in valid_paths.
  6. Log check: run 1 to 5 with log capture on, and assert that no value of a personal-data field appears in the bonsapi or MCP logs. Only ids, mask paths, sizes and reasons may appear.

11. Schema evolution (0.1.10) #

Rules for protobuf/external/ and protobuf/shared/:

  • Changes are additive only: new fields, new messages, new methods, new enum values.
  • Never change a field’s number, type, name or label (singular, optional, repeated), and never move a field into or out of a oneof.
  • Fields and enum values are never removed from a v1 package. A field that is no longer used gets [deprecated = true] and stops being filled. buf breaking uses the FILE rules, which reject any removal even when its number is reserved. FILE is kept, rather than the WIRE_JSON …_UNLESS_RESERVED rules, because it also rejects renames, and a rename breaks mask paths. Removal happens only in a new package version (v2).
  • An enum’s zero value is <ENUM>_UNSPECIFIED, and values are append-only.
  • A request field never becomes required later. New request fields are optional, and their default keeps the old behaviour.
  • A change that cannot be additive goes in a new package version (v2) served beside v1 until callers move.
  • Mask paths are part of the API. Renaming a field renames a path, so it is a breaking change even where the wire format would allow it.

CI checks (added in this ticket):

  • protobuf/buf.yaml configures buf lint (STANDARD rules) and buf breaking (FILE rules) for external/, shared/ and options/. internal/ is not linted, and gets only the WIRE breaking rules: tofu-extraction and other workers compile it separately (libs/rust/tofu-grpc/build.rs) and deploy on their own schedule, so its wire format must stay compatible, but its names are not an API.
  • mise run proto-lint-check runs buf lint, and is part of mise run check.
  • mise run proto-breaking-check runs buf breaking against origin/main, so locally it compares with the remote, not a stale local main (run git fetch origin main first). CI compares against the run’s own base commit (the pull request base, the merge queue base, or the commit before a push; skipped on releases), checked out in a separate worktree, which works on the job’s shallow clone. CI runs both checks in the Misc job whenever protobuf/** changes.
  • Existing files that predate these rules are exempt from the lint rules they break (see the ignore_only list in buf.yaml). New files get no exemptions, so every Phase 0 service is lint-clean from its first PR.
  • PACKAGE_DIRECTORY_MATCH and DIRECTORY_SAME_PACKAGE are off for everything. build-protobuf.rs requires protos to live under external/, internal/, shared/ or options/ with one package per file, which cannot match the bonsapi_grpc.* package paths.

Golden snapshots of default masks (built in 0.2 with the codegen, because no default exists before it): a unit test writes every read type’s normalised DEFAULT_MASK and its valid path list to apps/bonsapi/src/grpc/field_selection/snapshots/<read_type>.txt and compares them with the committed file. Any change to a default, or to the set of valid paths, shows up as a diff in review, so it is always deliberate.

12. Observability and SLOs (0.1.11) #

Traces. The MCP sends traceparent on every call (src/grpc/shared/metadata.ts). bonsapi already continues it (extract_parent_trace_context), so one trace runs from the MCP tool call through bonsapi to the database. Each RPC span carries rpc.system=grpc, rpc.service, rpc.method, rpc.grpc.status_code, tofu.error.reason, tofu.read_mask.path_count, tofu.read_mask.is_default, tofu.load_groups, tofu.batch.size and tofu.response.bytes, and never a field value.

Metrics (OpenTelemetry, beside HttpMetrics in model/app/metrics.rs):

Metric Type Attributes
grpc.server.duration histogram, ms service, method, code
grpc.server.requests counter service, method, code, reason
grpc.server.response.size histogram, bytes service, method
grpc.read_mask.paths counter, one per requested path service, method, path (from the generated PATHS list, unknown paths counted as _invalid, so cardinality is bounded)
grpc.read_mask.default counter service, method, is_default
grpc.quota.rejections counter service, kind (read or write)

The path counter shows which fields callers actually use. It is the evidence for adding a path to a default, or for keeping one there.

SLOs, per service, over 28 days:

  • Availability: 99.9% of calls do not end in INTERNAL, UNAVAILABLE or DEADLINE_EXCEEDED. Client errors (INVALID_ARGUMENT, PERMISSION_DENIED, NOT_FOUND, ABORTED, RESOURCE_EXHAUSTED) do not count against it. The error budget is 0.1%, about 40 minutes of full outage in 28 days.
  • Latency: p95 under 300 ms for Get and BatchGet, under 800 ms for List and Update.

Alerts (Datadog monitors on the metrics above, multi-window burn rate):

  • Page: the budget burns at 14.4× over both 1 hour and 5 minutes.
  • Ticket: the budget burns at 6× over both 6 hours and 30 minutes.
  • Ticket: p95 latency is over its SLO for 30 minutes.

13. Worked examples #

Examples are in proto JSON form. Values are made up.

Direct expense: default read #

// GetDirectExpense
{ "id": "6f0c…", "entity_id": "a1b2…" }          // no read_mask: default mask
// reply (about 1.3 KB for two lines)
{
  "id": "6f0c…", "entity_id": "a1b2…", "etag": "W/\"9c41e0…\"",
  "extraction_id": "3c4d…", "status": "READY", "publish_sync_status": "NOT_SYNCED",
  "data": {
    "id": "d9e1…", "action_type": "UPDATE",
    "title": "Taxi", "transaction_date": 1794700800000, "currency_id": "JPY",
    "total_amount": 2310, "line_amount_type": "INCLUSIVE", "contact_name": "Example Taxi",
    "line_items": [
      { "id": "l1…", "description": "Fare", "quantity": 1, "unit_price": 2100, "amount": 2100, "tax_amount": 210 },
      { "id": "l2…", "description": "Toll", "quantity": 1, "unit_price": 210, "amount": 210, "tax_amount": 0 }
    ]
  }
}

No metadata, no bboxes, no traces, no sync_activities: they are not in the default, so their queries do not run.

Invoice: batch get with an explicit mask and a heavy field #

// BatchGetInvoices
{
  "entity_id": "a1b2…",
  "ids": ["i1…", "i2…", "i3…"],
  "read_mask": "data.invoice_number,data.total_amount,data.metadata.traces"
}
// reply: per-item results in request order
{
  "results": [
    { "id": "i1…", "invoice": { "id": "i1…", "entity_id": "a1b2…", "etag": "…",
        "data": { "id": "…", "action_type": "CREATE", "invoice_number": "INV-001", "total_amount": 11000,
                  "metadata": { "traces": { "ap_bill_data.invoice_number": { … } } } } } },
    { "id": "i2…", "error": { "code": 5, "message": "Not found.",
        "details": [{ "@type": "type.googleapis.com/google.rpc.ErrorInfo",
                      "reason": "RESOURCE_NOT_FOUND", "domain": "bonsapi.gotofu.com",
                      "metadata": { "resource_type": "invoice" } }] } },
    { "id": "i3…", "invoice": { … } }
  ]
}

data.metadata.traces is heavy, so it comes back only because it was named. data.metadata.bboxes stays out.

Bank statement: header, then transactions by page #

// GetBankStatement: latest version with data, header only
{ "id": "b1…", "entity_id": "a1b2…", "version": { "latest": {} } }
// reply: default mask, data.transaction_count = 33, no transactions

// ListBankTransactions
{
  "bank_statement_id": "b1…", "entity_id": "a1b2…",
  "page_size": 50, "read_mask": "date,description,amount"
}
// reply
{ "transactions": [ { "id": "t1…", "date": 1794700800000, "description": "…", "amount": -1200 }, … ],
  "next_page_token": "" }

version is a oneof { Latest latest; LatestVerified latest_verified; string data_id; }. latest skips a version that is still being written, so a re-read never returns an empty version (the details are in 0.5).

// ListExtractions
{
  "entity_id": "a1b2…",
  "filter": "type = \"DIRECT_EXPENSE\" AND status = \"NEEDS_REVIEW\"",
  "order_by": "created_at desc",
  "page_size": 100,
  "read_mask": "id,status,document_id,direct_expense_id"
}
// reply (the caller holds DIRECT_EXPENSE:READ, so direct_expense_id is allowed)
{ "extractions": [ { "id": "e1…", "entity_id": "a1b2…", "etag": "…", "status": "NEEDS_REVIEW",
                     "document_id": "doc1…", "direct_expense_id": "6f0c…" }, … ],
  "read_time": "2026-11-18T02:40:00Z",
  "resume_time": "2026-11-18T02:35:00Z",
  "next_page_token": "eyJr…" }

Document: unknown path #

// GetDocument
{ "id": "doc1…", "entity_id": "a1b2…", "read_mask": "file_name,private_metadata" }
// error
{
  "code": 3, "message": "read_mask has 1 invalid path.",
  "details": [
    { "@type": "type.googleapis.com/google.rpc.ErrorInfo", "reason": "INVALID_READ_MASK",
      "domain": "bonsapi.gotofu.com",
      "metadata": { "invalid_paths": "private_metadata",
                    "valid_paths": "id,entity_id,file_name,mime_type,size,status,source,…" } },
    { "@type": "type.googleapis.com/google.rpc.BadRequest",
      "field_violations": [ { "field": "read_mask.paths[1]", "description": "Unknown path." } ] }
  ]
}

Direct expense: partial update with a stale etag #

// UpdateDirectExpenseData
{
  "id": "6f0c…", "entity_id": "a1b2…",
  "data": { "external_account_id": "acc-610", "line_items": [ { "id": "l2…", "external_account_id": "acc-620" } ] },
  "update_mask": "data.external_account_id,data.line_items.external_account_id",
  "etag": "W/\"9c41e0…\"",
  "request_id": "0b7e…",
  "read_mask": "data.external_account_id,data.line_items.external_account_id"
}
// error: someone saved in between
{
  "code": 10, "message": "The direct expense changed since it was read.",
  "details": [
    { "@type": "type.googleapis.com/google.rpc.ErrorInfo", "reason": "ETAG_MISMATCH",
      "domain": "bonsapi.gotofu.com", "metadata": { "current_etag": "W/\"77ab02…\"" } },
    { "@type": "type.googleapis.com/bonsapi_grpc.direct_expense.v1.ConflictDetail",
      "current_etag": "W/\"77ab02…\"",
      "current": { "id": "6f0c…", "etag": "W/\"77ab02…\"", "data": { "external_account_id": "acc-600", … } } }
  ]
}

The caller merges with current and retries with the new etag and a new request_id. ETAG_MISMATCH is a stored outcome (§7), so resending the old id would replay this error.

14. Open questions for review #

For the MCP owner and the bonsapi owners (0.1.4). Approval is a comment on ENG-9842 from two engineers, including the MCP owner.

  1. Should the bank statement default include the first page of transactions, rather than none?
  2. Is a required etag on updates acceptable for the MCP write tools? It means a read before every write, which the tools mostly do already.
  3. The quota starting values (600 read and 120 write units per organisation per minute).
  4. Should ListExtractions and ListDocuments accept a filter across entities for organisation-wide reads, or stay one entity per call?