gRPC read contract: field selection for bonsapi #
Classification: Restricted (describes source code and access control).
INTERNAL - DO NOT FORWARD
Status: draft for review (ENG-9842, Phase 0 of ENG-9713). Every new external gRPC service in Phase 0 (0.3 to 0.7) follows this page. A change to a rule here is a change to this page first.
Why #
Callers cannot choose which fields a read returns, so the MCP strips them itself (withoutBboxes, capLineItems, capBytes, keepBboxes) and decides the reply shape in the wrong layer. With this contract the caller names the fields it wants, bonsapi loads and returns only those, and the MCP passes the caller’s choice straight through.
Scope: the new services for direct expenses, extractions, documents, bank statements and invoices (AP bill and AR invoice) under protobuf/external/. The existing entity, extraction status, knowledge and pricing services are not changed by this page. Webapps never call these services, and no REST route, REST response or OpenAPI entry changes (see the guardrail in ENG-9841).
1. Services and naming #
One service per resource, one package per service, versioned from the start:
| Resource | Proto file | Package | Service |
|---|---|---|---|
| Direct expense | external/direct_expense/direct_expense.proto |
bonsapi_grpc.direct_expense.v1 |
DirectExpenseService |
| Extraction | external/extraction/extraction.proto |
bonsapi_grpc.extraction.v1 |
ExtractionService |
| Document | external/document/document.proto |
bonsapi_grpc.document.v1 |
DocumentService |
| Bank statement | external/bank_statement/bank_statement.proto |
bonsapi_grpc.bank_statement.v1 |
BankStatementService |
| Invoice (AP bill, AR invoice) | external/invoice/invoice.proto |
bonsapi_grpc.invoice.v1 |
InvoiceService |
Method names follow the AIP standard methods so callers can guess them:
| Method | Shape |
|---|---|
Get<Resource> |
Get<Resource>Request { string id; string entity_id; FieldMask read_mask; } |
BatchGet<Resources> |
ids plus one read_mask, per-item results (§8) |
List<Resources> |
request: entity_id, page_size, page_token, filter, order_by, read_mask. Reply: the resources, next_page_token, read_time, resume_time (§8) |
List<Resource>Versions |
the append-only data history of one resource, newest first |
Update<Resource>Data |
partial update with update_mask, etag, request_id, read_mask (§5) |
Reserved names. Watch<Resources> (server streaming, for example rpc WatchExtractions(WatchExtractionsRequest) returns (stream WatchExtractionsResponse)) is reserved for the live status stream that tofu-web needs (ENG-9891). It is not built in Phase 0, and no other method may take a Watch prefix.
Resource shape. Each resource message is the parent record plus the data version a read selected, under data, with lines under data.line_items (data.transactions for bank statements). Field names match the REST response names so callers moving off REST keep their vocabulary. Bookkeeping metadata is a typed message, never a JSON string, so mask paths can reach into it (data.metadata.traces).
Explicit presence. Every scalar on a resource message is optional, so “not selected” (unset) is never confused with a real zero, empty string or false. A field that was selected but left out because the caller may not read it is listed in omitted_mask (§2), so an unset field that is not in omitted_mask really has no value: an unset link id means there is no link.
2. Read masks #
Every read request has a google.protobuf.FieldMask read_mask (AIP-157, partial responses).
Paths #
- Paths are proto field names in
snake_case, separated by dots, relative to the resource message. On aListorBatchGet, paths are relative to each returned resource, not to the response (data.total_amount, neverdirect_expenses.data.total_amount). - A path through a repeated message field selects that subfield on every element:
data.line_items.description. - A path that ends at a message field selects every non-heavy field under it:
datareturns the whole data version except its heavy fields. - Map fields are selected whole. There are no map-key paths (
data.metadata.tracesyes,data.metadata.traces.titleno). *alone selects every non-heavy field of the resource. It cannot be combined with a prefix (data.*is invalid; writedata).- Heavy fields (
traces,bboxes, page coordinates, sync activity payloads) are returned only when their full path is named. Neither*nor a parent path expands to them. - Duplicate and overlapping paths are allowed and normalised (
data,data.titleis the same asdata).
Always returned #
id, entity_id and etag are always set, whatever the mask, so a caller can always correlate and write back. On a selected data version, data.id and data.action_type are always set too.
Every resource also carries google.protobuf.FieldMask omitted_mask. It is set whenever a path selected through *, a parent path or the default mask was left out because the caller lacks its permission (§10), and it lists those paths. It is empty when nothing was left out.
Empty mask #
An omitted or empty mask means the default mask of that read type (§3). There is no way to ask for “nothing”.
Invalid masks #
Every bad mask fails the whole call with INVALID_ARGUMENT, a BadRequest.FieldViolation per bad path (read_mask.paths[2]), and the paths this caller may use in ErrorInfo.metadata["valid_paths"]. Nothing is silently ignored.
- A path that does not exist on the resource, or a
*misuse: reasonINVALID_READ_MASK. - A path that exists but the caller may not read (§10): reason
FIELD_NOT_ALLOWED. - A mask over the limits in §9: reason
READ_MASK_TOO_LARGE.
valid_paths is computed for the caller, so it never lists a path they may not read.
Pushdown #
Each path maps to a load group, and bonsapi only runs the queries for the groups a mask touches:
| Load group | Paths | What is not loaded when the group is not selected |
|---|---|---|
header |
parent and data.* scalars |
always loaded (ids, status, etag) |
metadata |
data.metadata.* |
the metadata jsonb is read with heavy keys removed in SQL (metadata - 'traces' - 'bboxes') unless they were named |
lines |
data.line_items.* / data.transactions.* |
the line query is skipped |
line_metadata |
data.line_items.metadata.* |
per-line metadata jsonb, same rule as metadata |
sync |
data.sync_activities.* |
the sync activity query is skipped |
creator |
data.creator.* |
the user lookup is skipped |
pages |
pages.* (document, extraction) |
the page query is skipped |
private_metadata is never part of any read type, so the new read queries never select it. It is the largest column on the measured tables (medians: 15.2 KB per bank statement data row, 10.1 KB per invoice data row, 17.9 KB per document page row), so not loading it is the main saving on the database side.
3. Default masks and sizes #
Defaults hold ids, status and the main bookkeeping fields: enough to review, match and publish a record, nothing about where a value was found. Heavy fields are never in a default.
Defaults per read type #
Direct expense (GetDirectExpense, ListDirectExpenses, BatchGetDirectExpenses)
id, entity_id, extraction_id, status, external_id, publish_sync_status, last_publish_at,
data.title, data.reference, data.transaction_date, data.currency_id, data.total_amount,
data.line_amount_type, data.spend_type, data.payment_type, data.contact_name,
data.external_contact_id, data.external_account_id, data.external_tax_rate_id,
data.line_items.id, data.line_items.description, data.line_items.quantity,
data.line_items.unit_price, data.line_items.amount, data.line_items.tax_amount,
data.line_items.external_account_id, data.line_items.external_tax_rate_id
Invoice (AP bill and AR invoice)
id, entity_id, extraction_id, type, status, external_id, publish_sync_status, last_publish_at,
data.title, data.invoice_number, data.invoice_reference, data.invoice_date, data.due_date,
data.currency_id, data.total_amount, data.line_amount_type, data.contact_name,
data.external_contact_id, data.external_tax_rate_id,
data.line_items.id, data.line_items.description, data.line_items.quantity,
data.line_items.unit_price, data.line_items.tax_amount, data.line_items.external_account_id,
data.line_items.external_tax_rate_id, data.line_items.external_item_id
Bank statement (statement header only; transactions are read with ListBankTransactions, which pages, so a 300-line statement never needs one huge reply). status is the extraction’s status, because the bank statement row has none.
id, entity_id, extraction_id, status, data.external_id,
data.bank_name, data.account_number, data.external_account_id, data.currency_id,
data.opening_balance, data.closing_balance, data.start_date, data.end_date, data.transaction_count
ListBankTransactions default: id, date, description, amount, reference, currency_id, external_contact_id, external_account_id, external_tax_rate_id.
Extraction
id, entity_id, type, status, title, error_phase, error_message, source, created_at, updated_at,
document_id, direct_expense_id, invoice_id, bank_statement_id
The link fields (document_id and the record ids) are what 1.2 needs to pair a document with its draft without guessing by creation time. Each record id needs the read permission of the record it points to (§10). A caller without it gets the extraction with that id unset and listed in omitted_mask.
Document
id, entity_id, file_name, mime_type, size, status, source, extraction_types, page_count,
extraction_ids, created_at, updated_at
Sizes #
Measured on stored rows: aggregate medians across many organisations, computed in SQL, with no records read out. The default column is the JSON size of exactly the default fields’ stored values. Derived fields (sync status, last publish time, link ids, counts) are not stored on these rows and add under 200 bytes. Medians of parts do not add up exactly, so the full-size totals are indicative.
| Read type | Sample | Full stored size (median parts) | Default mask (measured, median) |
|---|---|---|---|
| Direct expense, 2 lines | 317 versions, 21 entities, last 14 days | header 1.7 KB + metadata 2.0 KB (traces 1.6 KB, bboxes 0.4 KB) + lines 2.0 KB + line metadata 2.7 KB ≈ 8.4 KB, before REST adds the extraction, sync activities and creator (about 20,000 characters as the partner received it) | 1.33 KB (p90 1.40 KB) |
| Invoice, 2 lines | 347 versions, last day | header 9.1 KB (mostly private_metadata) + traces 3.3 KB + bboxes 1.7 KB + lines 2.6 KB + line metadata 5.9 KB |
1.27 KB (p90 1.38 KB) |
| Bank statement | 1,935 versions, last 14 days, median 33 transactions | header 16.1 KB (private_metadata 15.2 KB) + metadata 3.5 KB + 33 × (0.9 KB + 2.2 KB metadata) ≈ 122 KB |
header 0.39 KB; 0.29 KB per transaction (p90 0.34 KB), so a page of 50 is about 15 KB |
| Extraction | 2,000 rows, last day | 0.4 KB row | 0.30 KB |
| Document | 2,000 rows, last day, median 1 page | 0.7 KB row + 15.1 KB per page (private_metadata 17.9 KB) |
0.29 KB |
The QA target is about 3,000 characters for a two-line receipt. The direct expense default measures 1.33 KB at the median and 1.40 KB at p90, under half the target. Proto JSON adds little to these sizes, and the binary wire format is smaller. 0.9’s reply size budget tests measure the real serialised replies and fail if a default grows past its budget.
Changing a default safely #
- Adding a path to a default is allowed in any PR, as long as the size budget test still passes. Callers that want a fixed shape send an explicit mask.
- Removing a path from a default is a deprecation. Announce it in the MCP changelog and the PR, keep it for at least 30 days, and update the golden snapshot in its own PR.
- Every default lives in a golden snapshot (§11), so no default changes by accident.
4. How masks are applied (0.1.2) #
Decision: proto field options plus generated code. Not runtime reflection, not hand-written matching.
A new extension-only proto, protobuf/options/field_selection.proto, sits next to log_redaction.proto:
package bonsai.options.v1; // new files get no lint exemptions, so the package is versioned
message FieldSelection {
bool read_default = 1; // in this read type's default mask
bool heavy = 2; // only returned when its full path is named
string load_group = 3; // §2 pushdown group; inherited from the parent when empty
string read_permission = 4; // extra permission beyond the resource read, e.g. "MEMBER:READ"
bool writable = 5; // may appear in an update_mask
}
extend google.protobuf.FieldOptions {
optional FieldSelection field_selection = 50001;
}
build-protobuf.rs already compiles every proto under protobuf/. It gains a step that asks tonic_prost_build for the file descriptor set, decodes it with prost-reflect as a build-dependency only (plain prost-types drops custom options, because prost discards unknown extension fields), and writes one module per resource into src/grpc/__generated/, committed like the rest of that directory:
PATHS: every valid path with its flags, load group and permission.DEFAULT_MASK: the normalised default.fn load_plan(&FieldMask) -> Result<LoadPlan, MaskError>: validates, expands*and parent paths, drops heavy fields unless named, returns the load groups to query.fn prune(&mut Resource, &NormalisedMask): clears every field not in the mask, recursively, including inside repeated fields.
Why this option:
- Paths are checked at build time against the real messages, so a renamed field breaks the build, not a caller.
- No descriptor pool or dynamic messages at runtime, and no new runtime dependency.
- It is the same pattern as
log_safe: the rule lives next to the field, so the default mask, the permission and the redaction status of a field are read in one place. prost-reflectat runtime was rejected: it would need a descriptor set loaded at start-up andDynamicMessageconversions on every reply, for no gain over generated code. Hand-written matching was rejected because it drifts from the schema.
The TS side reads the same option, the way scripts/generate-log-safe-fields.ts reads log_safe, so the MCP can describe valid fields values in tool schemas without a second list.
5. Writes #
Update<Resource>Data (0.7) is a partial update that writes a new version in the append-only data history:
message UpdateDirectExpenseDataRequest {
string id = 1;
string entity_id = 2;
DirectExpenseData data = 3; // only the update_mask paths are read
google.protobuf.FieldMask update_mask = 4; // required, explicit paths only, no "*"
string etag = 5; // required, from the last read (§7)
string request_id = 6; // required UUID (§7)
google.protobuf.FieldMask read_mask = 7; // shape of the reply; empty = default mask
}
update_maskis required and lists explicit paths. Only fields markedwritableare allowed, otherwiseINVALID_ARGUMENT, reasonINVALID_UPDATE_MASK.- A field in
update_maskbut unset indatais cleared. A field set indatabut not inupdate_maskis ignored. data.line_itemsin the mask replaces the whole line list. A line that carries an existingidkeeps its stored metadata (bboxes and traces). New lines have noid.data.line_items.<field>updates that field only on the lines sent, matched byid. Lines not sent are unchanged, and an unknown lineidisINVALID_ARGUMENT.data.line_itemsand anydata.line_items.<field>in the sameupdate_maskcontradict each other (replace the list or patch it), so that mask isINVALID_UPDATE_MASK. The same rule applies to any message path listed with one of its own subpaths. Duplicate identical paths are allowed.- Heavy fields are never writable. The server carries
bboxesandtracesforward for every field the update did not change, which moves the MCP’skeepBboxeslogic into bonsapi. - The reply is the updated resource with
read_maskapplied. - The gRPC write calls new service functions. The REST write path and the
is_extraction_updatingfield the webapp waits on keep their behaviour.
6. Errors (0.1.5) #
Every error is a google.rpc.Status: a code, a message for humans, and typed details (bonsapi uses tonic-types to attach them):
google.rpc.ErrorInfo, always present:reason(from the table below),domain=bonsapi.gotofu.com,metadatawith string keys only.google.rpc.BadRequestwithFieldViolations for everyINVALID_ARGUMENTcaused by a request field.google.rpc.RetryInfoon every retryable error.google.rpc.RequestInfo: the request id, so support can find the trace.
Stable reasons. A reason is an API. It is never renamed or reused, and callers branch on reason, never on the message.
| Code | Reason | When | Metadata |
|---|---|---|---|
INVALID_ARGUMENT |
INVALID_READ_MASK |
unknown path or * misuse |
invalid_paths, valid_paths |
INVALID_ARGUMENT |
FIELD_NOT_ALLOWED |
a path that was named explicitly needs a permission the caller lacks | invalid_paths, valid_paths (only paths this caller may read) |
INVALID_ARGUMENT |
INVALID_UPDATE_MASK |
unknown, non-writable, missing, or overlapping path | invalid_paths, writable_paths |
INVALID_ARGUMENT |
READ_MASK_TOO_LARGE |
over the path count or depth limit | max_paths, max_depth |
INVALID_ARGUMENT |
INVALID_PAGE_TOKEN |
malformed, expired, or issued for another filter or order | |
INVALID_ARGUMENT |
INVALID_FILTER / INVALID_ORDER_BY |
syntax error or a field that is not allow-listed | allowed_fields |
INVALID_ARGUMENT |
BATCH_TOO_LARGE |
more ids than the batch limit | max_ids |
INVALID_ARGUMENT |
RESPONSE_TOO_LARGE |
one resource alone exceeds the size limit (§9). On BatchGet this is a per-item error |
size_bytes, limit_bytes |
INVALID_ARGUMENT |
REQUEST_ID_REUSED |
same request_id, different request |
|
INVALID_ARGUMENT |
ETAG_REQUIRED |
a write without etag |
|
UNAUTHENTICATED |
UNAUTHENTICATED |
missing or invalid token | |
PERMISSION_DENIED |
ENTITY_ACCESS_DENIED |
no read on this entity or resource type | resource_type |
NOT_FOUND |
RESOURCE_NOT_FOUND |
no such id in this entity (also used when the resource is in another entity, so ids cannot be probed) | resource_type |
ABORTED |
ETAG_MISMATCH |
the resource changed since the caller’s read | current_etag, plus a ConflictDetail (§7) |
ABORTED |
REQUEST_IN_PROGRESS |
a duplicate of a write that is still running | RetryInfo |
INVALID_ARGUMENT |
QUOTA_COST_TOO_HIGH |
one call costs more units than the whole bucket holds, so waiting cannot help | cost, capacity |
RESOURCE_EXHAUSTED |
QUOTA_EXCEEDED |
over the organisation’s quota | quota, RetryInfo |
DEADLINE_EXCEEDED |
DEADLINE_EXCEEDED |
the server deadline passed | |
UNAVAILABLE |
UPSTREAM_UNAVAILABLE |
database, cache or accounting provider not reachable | RetryInfo |
INTERNAL |
INTERNAL |
anything else | request_id only |
Per-item results. Batch calls never fail as a whole because one item failed. Each item has its own result in request order, oneof { Resource resource; google.rpc.Status error; }. Only request-level problems (bad mask, too many ids, quota) fail the call.
Nothing internal leaves the service. No stack traces, SQL, table or column names, internal service names, file paths or upstream error bodies go into a message or into metadata. INTERNAL carries only the request id; the details stay in the logs, joined by trace id. The existing map_api_error (grpc/auth.rs) currently puts err.to_string() into INTERNAL. The new services use a new mapper that follows this table, and map_api_error stays as it is for the existing services.
7. Concurrency and idempotency (0.1.6) #
Etags. Every resource returns an opaque etag. It is a hash of everything a write or a status change can alter, not of a timestamp:
- the id of the latest data version (every data edit appends a new version);
- every column of the parent row except
created_atandupdated_at, so a parent column added later is covered without a spec change; - the linked extraction’s
status, which changes on mark ready, done and draft without a new data version; - the id of the latest sync activity, which changes on publish and import without a new data version.
What that covers today, per type:
| Type | Parent columns in the hash | Also hashed |
|---|---|---|
| Direct expense | extraction_id, external_id, source, is_ai_autosplit, status |
latest data id, extraction status, latest sync activity id |
| Invoice | extraction_id, external_id, status, source, is_ai_autosplit, last_publish_at, last_import_at, publish_error_type, import_error_type, publish_sync_status, import_sync_status |
latest data id, extraction status, latest sync activity id |
| Bank statement | extraction_id, is_ai_autosplit, metadata, version (no status or external_id on this row: status is the extraction’s, and external_id is on the data version) |
latest data id, extraction status, latest sync activity id |
| Extraction | every extraction column | none |
| Document | every document column | none |
The stored updated_at column is not used, because nothing guarantees every status or sync change bumps it.
changed_at. For the same reason, the API’s updated_at (returned, filtered and sorted on) is backed by a new changed_at column on each parent table, not by the stored updated_at. Database triggers bump it in the same transaction as any change to an etag input: the parent row, a new data version, the extraction’s status, or a new sync activity. So whenever the etag changes, updated_at moves forward too, and a publish or mark-done shows up in a “changes since” read. It is a new column, so the REST responses and the webapp keep reading the old one unchanged. 0.2 adds the column, its backfill and its index. Callers treat the etag as an opaque string. It is computed in the same query that loads the header group, so it costs no extra query.
Preconditions. Every write takes the etag the caller last read and is required on updates (ETAG_REQUIRED without it). The etag is recomputed inside the write transaction, after SELECT … FOR UPDATE on the parent row, so two writers cannot both pass the check. If it no longer matches, the write is refused with ABORTED / ETAG_MISMATCH, and the details carry a ConflictDetail. The caller can then merge and retry without another read.
Each service package defines its own ConflictDetail, so the current resource is typed:
// bonsapi_grpc.direct_expense.v1, and the same per service.
message ConflictDetail {
string current_etag = 1;
// The resource as it is now, with the request's read_mask and the caller's
// field permissions applied (§10).
DirectExpense current = 2;
}
Request ids. Every write takes a client-generated UUID request_id. bonsapi stores (organization_id, user_id, method, request_id) → hash of the request and the outcome, for 24 hours. The user id is in the key so two users of one organisation can never collide on an id.
- Stored outcomes are only final ones: a success, or a refusal that the same request would get again (
INVALID_ARGUMENT,PERMISSION_DENIED,NOT_FOUND,ABORTED/ETAG_MISMATCH). - Transient failures are not stored (
UNAVAILABLE,DEADLINE_EXCEEDED,INTERNAL,RESOURCE_EXHAUSTED). The write’s own transaction has rolled back and the claim is released, or its lease lapses (below), so a retry with the same id runs the write again. This is the timeout case request ids exist for. - Same id and same request, with a stored outcome: that outcome is returned and nothing is applied a second time.
- Same id and a different request:
INVALID_ARGUMENT/REQUEST_ID_REUSED. - Same id while the first call still holds a live claim:
ABORTED/REQUEST_IN_PROGRESS, withRetryInfoset to when the lease ends.
Claim and lease. The claim and the outcome are written at different times:
- The claim is committed in its own short transaction before the write starts, so a concurrent duplicate can see it. It carries a lease that ends at the method’s server deadline (§9) plus 5 seconds.
- The outcome is written inside the write’s own transaction, so a write is never applied without its outcome, and an outcome is never stored for a write that rolled back.
- If the process dies mid-write, the claim has no outcome and its lease lapses within seconds, not 24 hours. The next retry takes over the claim and runs the write, because the rolled-back write applied nothing.
- A retry first looks for a stored outcome, then for a live lease. So a write that committed just before a crash is answered from its outcome and never applied twice.
Reads are naturally idempotent and take no request id.
8. Pagination, batch get, filter and sort (0.1.7) #
Keyset pagination. page_size defaults to 20 and is capped at 100. A larger value is lowered to 100; this is standard AIP-158 behaviour and is not a silent cut, because next_page_token says there is more. page_token is opaque: a signed, base64url token holding the last row’s sort key and id, the read_time (below), and a hash of filter and order_by. Every order ends with id as a tiebreak. An empty next_page_token means the last page.
Both sort keys are stable within one paging run:
created_atnever changes. Rows created after the first page are excluded byread_time, so pages never repeat a row.updated_at(backed bychanged_at, §7) changes whenever the etag changes. The first page recordsread_time(the server time when that page is read), and every later page addsupdated_at <= read_time. A row edited during the run therefore moves aboveread_timeand leaves the result. It never appears twice, and the order of the rest never shifts.
Late commits. Both timestamps are stamped when a write’s transaction runs, not when it commits. A row stamped just below read_time whose transaction commits after the cursor has passed its position is missed by the run. So every reply also returns resume_time = read_time minus a 5-minute commit-lag margin:
- After the last page, a caller that needs everything lists again with
filter: "updated_at > <resume_time>"(orcreated_at). That is the incremental “changes since” read tofu-web needs. - The overlap window returns some rows a second time, so callers dedupe by
idandetag. The same pair is the same version and is skipped. - The margin must be longer than any transaction that stamps these columns. 0.2 checks that the database’s statement and idle-in-transaction timeouts keep every such transaction below it, and raises the margin if not.
A token used with a different filter or order is INVALID_PAGE_TOKEN. There is no total_size, because counts on large entities are expensive.
Batch get. BatchGet<Resources>Request { string entity_id; repeated string ids; FieldMask read_mask; } takes 1 to 100 distinct ids, one mask for all of them, and returns per-item results (§6) in request order. If the results together pass the size limit, items are filled in request order. The first item that no longer fits, and every item after it, gets a per-item RESPONSE_TOO_LARGE error, so the caller re-requests exactly those ids. A duplicated id fails the call with INVALID_ARGUMENT.
Filter. An AIP-160 subset: field op value comparisons joined by AND, with OR only between values of the same field. Operators: =, !=, <, <=, >, >=. Values are quoted strings, numbers, booleans or enum names. Only allow-listed fields can be filtered, per list method, and each one is backed by an index. Field permissions (§10) apply to filter and order_by as they do to masks: naming a field the caller may not read (for example extraction_id without INVOICE:READ) fails with INVALID_ARGUMENT / FIELD_NOT_ALLOWED, so a filter can never probe a hidden value. The allowed_fields in INVALID_FILTER errors are computed for the caller in the same way.
| List | Filter fields |
|---|---|
ListDirectExpenses, ListInvoices, ListBankStatements |
status, created_at, updated_at, extraction_id, publish_sync_status, data.action_type, data.currency_id (plus type for invoices) |
ListExtractions |
type, status, created_at, updated_at, document_id |
ListDocuments |
status, source, created_at, updated_at |
Example: status = "READY" OR status = "DONE" AND created_at >= 1794787200000. OR binds tighter than AND here, as in AIP-160.
Reply size on lists. A List returns as many rows as fit in the size limit (§9), in order. If the page fills before page_size rows, it ends early with a next_page_token, so nothing is skipped and the reply says there is more. Only a single row that is over the limit on its own fails the call, with RESPONSE_TOO_LARGE and that row’s id in the metadata.
Sort. order_by is one allow-listed field with an optional desc: created_at (default created_at desc) or updated_at. id is always added as the last key. Only one sort field is allowed, because the read_time rule above relies on a single key.
9. Limits and resilience (0.1.8) #
| Limit | Value | Over the limit |
|---|---|---|
Server deadline: Get, BatchGet |
5 s, 10 s | DEADLINE_EXCEEDED |
Server deadline: List, List…Versions |
10 s | DEADLINE_EXCEEDED |
Server deadline: Update… |
15 s | DEADLINE_EXCEEDED |
| Page size | default 20, max 100 | lowered to 100 |
| Batch ids | 100 | BATCH_TOO_LARGE |
| Mask paths | 64 | READ_MASK_TOO_LARGE |
| Mask depth | 5 segments | READ_MASK_TOO_LARGE |
| Path length | 256 characters | INVALID_READ_MASK |
| Reply size | 1 MiB, or less if the caller sends tofu-max-response-bytes metadata |
Get: RESPONSE_TOO_LARGE. List: a shorter page with a next_page_token (§8). BatchGet: per-item RESPONSE_TOO_LARGE on the items that did not fit (§8). Never a cut resource. |
The server uses the shorter of the caller’s grpc-timeout and its own deadline. The reply size is checked after pruning and before sending. Errors name the size and the limit, so the caller can narrow the mask or the page size. A tofu-max-response-bytes value above 1 MiB is lowered to 1 MiB. The MCP sends its own budget (today DEFAULT_MAX_RESPONSE_BYTES, 200,000) in that metadata instead of cutting the reply itself.
Retries. Callers retry only:
- reads, on
UNAVAILABLE,DEADLINE_EXCEEDEDandABORTED/REQUEST_IN_PROGRESS; - writes that carry a
request_id, on the same codes; - any call that returns
RESOURCE_EXHAUSTEDwithRetryInfo, after the delay it gives.
Backoff is exponential with full jitter: base 100 ms, factor 2, cap 2 s, at most 3 attempts, always within the caller’s own deadline. INVALID_ARGUMENT, PERMISSION_DENIED, NOT_FOUND and ABORTED / ETAG_MISMATCH are never retried as they are.
Per-organisation quotas. A Redis token bucket per organisation, per service: 600 read units and 120 write units per minute. Costs:
| Call | Bucket | Units |
|---|---|---|
Get, List…Versions page |
read | 1 |
BatchGet |
read | 1 per id |
List |
read | 1 per 20 rows of the requested page_size, rounded up (a page of 100 costs 5) |
Update… |
write | 1 |
Units are charged before the call runs. A call that costs more than the whole bucket holds is INVALID_ARGUMENT / QUOTA_COST_TOO_HIGH, never a RESOURCE_EXHAUSTED that no wait could clear. This sits on top of the MCP’s per-user limit (see Rate Limiting), so one organisation with many users cannot starve the others. Over quota is RESOURCE_EXHAUSTED / QUOTA_EXCEEDED with RetryInfo set to when the bucket refills. Like the MCP limiter, it fails open if Redis is down, and the gap is logged and counted. The numbers are starting values, to be set from the 0.9 metrics before tofu-web uses these services.
10. Field-level authorization and privacy (0.1.9) #
Resource permissions. Every call first checks the same entity-scoped resource read that REST uses for that resource (grpc/auth.rs → AccessControl):
| Service | Read | Write |
|---|---|---|
DirectExpenseService |
DIRECT_EXPENSE:READ |
DIRECT_EXPENSE:UPDATE |
InvoiceService |
INVOICE:READ |
INVOICE:UPDATE |
BankStatementService |
BANK_STATEMENT:READ |
BANK_STATEMENT:UPDATE |
ExtractionService |
INVOICE:READ (as REST extractions) |
n/a |
DocumentService |
DOCUMENT:READ |
n/a |
Fields that need more than the resource read.
| Paths | Extra permission | Why |
|---|---|---|
data.creator.*, data.sync_activities.creator.* |
MEMBER:READ |
names another user of the organisation |
data.sync_activities.* |
INTEGRATION:READ |
shows integration activity and provider error types |
pages.* on extraction |
DOCUMENT:READ |
page geometry and file references are document data |
extraction_id on direct expense, invoice and bank statement |
INVOICE:READ |
links to the extraction, which REST gates on INVOICE |
direct_expense_id on extraction |
DIRECT_EXPENSE:READ |
links to a record the caller may not read |
invoice_id on extraction |
INVOICE:READ |
same; always held by an extraction reader today, but stated so the rule does not depend on that |
bank_statement_id on extraction |
BANK_STATEMENT:READ |
same |
extraction_ids on document |
INVOICE:READ |
same, as REST maps extractions to INVOICE |
publish_sync_status and last_publish_at need only the resource read. They summarise the record’s own publish state, as REST already returns with that read, and carry no provider detail. That detail lives in data.sync_activities.*, which is gated above.
Never exposed on any path: private_metadata, user email addresses, storage file paths, signed URLs (these keep their own RPC), and provider credentials or raw provider responses.
A mask can never widen access. Paths are resolved against the caller’s permissions after the mask is validated:
- A forbidden path that was named explicitly fails the call with
INVALID_ARGUMENT/FIELD_NOT_ALLOWED, which lists the forbidden paths and the paths this caller may read. This is the same error shape as an unknown path, as the ticket specifies, so a caller has one way to correct a mask. - A forbidden path reached through
*, a parent path or the default mask is left out and listed inomitted_mask, so a default read never fails because of a field the caller did not ask for, and the caller can still tell “not permitted” from “no value”. - Pruning runs on the final reply, after the load plan, so a field loaded for another reason (for example to compute the etag) cannot leak.
- On batch and list calls, a resource in an entity the caller cannot read is a per-item
NOT_FOUND(batch) or is never matched (list).
Personal-data allow-list. These fields can hold personal data and are never log_safe: title, description, reference, contact_name, bill_address, ship_address, ship_from_address, tax_identification_number, learning, invoice_number, invoice_reference, account_number, bank_name, transaction description, contact_name, reference and check_number, metadata.traces (raw text read from the document), metadata.invoice_lines_aggregated_description, file_name, error_message, creator.*. A new field is personal data unless the PR argues otherwise. The 0.9 field catalog marks each field.
Logging and log_safe. Masks do not change redaction:
- Redaction stays per field. A field is logged verbatim only if it carries
(bonsai.options.log_safe) = true. A masked reply is a subset of the full message, so the same allow-list covers it. read_maskandupdate_maskpaths are field names, not values, so they arelog_safeand are logged. This is what makes the mask-usage metrics in §12 possible.- bonsapi does not log reply payloads, only sizes, path counts and error reasons. The MCP’s
redact.tskeeps applying the allow-list generated from the protos (scripts/generate-log-safe-fields.ts). - Being in a default mask does not make a field log-safe.
read_defaultandlog_safeare independent options.
Test plan with a restricted user (implemented in 0.3 to 0.6, one fixture per service):
- A user with a role that has the entity but not the resource read gets
PERMISSION_DENIED/ENTITY_ACCESS_DENIED, onGet,BatchGetandList. - A user with resource read but no
MEMBER:READ:- a default read succeeds, with
data.creatorunset and listed inomitted_mask; read_mask: "*"succeeds, withdata.creatorunset and listed inomitted_mask;read_mask: "data.creator.name"fails withINVALID_ARGUMENT/FIELD_NOT_ALLOWED,invalid_paths = data.creator.name, andvalid_pathsdoes not contain anydata.creatorpath.
- a default read succeeds, with
- A user with extraction read but no
DIRECT_EXPENSE:READ: a defaultGetExtractionsucceeds withdirect_expense_idunset and listed inomitted_mask, andread_mask: "direct_expense_id"fails withFIELD_NOT_ALLOWED. - A user with access to entity A only:
BatchGetwith ids from A and B returns resources for A and per-itemNOT_FOUNDfor B.Liston B fails withENTITY_ACCESS_DENIED. - A path to
private_metadata, or to any field that is not in the schema, isINVALID_READ_MASK, and its name does not appear invalid_paths. - Log check: run 1 to 5 with log capture on, and assert that no value of a personal-data field appears in the bonsapi or MCP logs. Only ids, mask paths, sizes and reasons may appear.
11. Schema evolution (0.1.10) #
Rules for protobuf/external/ and protobuf/shared/:
- Changes are additive only: new fields, new messages, new methods, new enum values.
- Never change a field’s number, type, name or label (singular,
optional,repeated), and never move a field into or out of aoneof. - Fields and enum values are never removed from a
v1package. A field that is no longer used gets[deprecated = true]and stops being filled.buf breakinguses the FILE rules, which reject any removal even when its number isreserved. FILE is kept, rather than theWIRE_JSON…_UNLESS_RESERVEDrules, because it also rejects renames, and a rename breaks mask paths. Removal happens only in a new package version (v2). - An enum’s zero value is
<ENUM>_UNSPECIFIED, and values are append-only. - A request field never becomes required later. New request fields are optional, and their default keeps the old behaviour.
- A change that cannot be additive goes in a new package version (
v2) served besidev1until callers move. - Mask paths are part of the API. Renaming a field renames a path, so it is a breaking change even where the wire format would allow it.
CI checks (added in this ticket):
protobuf/buf.yamlconfiguresbuf lint(STANDARD rules) andbuf breaking(FILE rules) forexternal/,shared/andoptions/.internal/is not linted, and gets only theWIREbreaking rules: tofu-extraction and other workers compile it separately (libs/rust/tofu-grpc/build.rs) and deploy on their own schedule, so its wire format must stay compatible, but its names are not an API.mise run proto-lint-checkrunsbuf lint, and is part ofmise run check.mise run proto-breaking-checkrunsbuf breakingagainstorigin/main, so locally it compares with the remote, not a stale localmain(rungit fetch origin mainfirst). CI compares against the run’s own base commit (the pull request base, the merge queue base, or the commit before a push; skipped on releases), checked out in a separate worktree, which works on the job’s shallow clone. CI runs both checks in the Misc job wheneverprotobuf/**changes.- Existing files that predate these rules are exempt from the lint rules they break (see the
ignore_onlylist inbuf.yaml). New files get no exemptions, so every Phase 0 service is lint-clean from its first PR. PACKAGE_DIRECTORY_MATCHandDIRECTORY_SAME_PACKAGEare off for everything.build-protobuf.rsrequires protos to live underexternal/,internal/,shared/oroptions/with one package per file, which cannot match thebonsapi_grpc.*package paths.
Golden snapshots of default masks (built in 0.2 with the codegen, because no default exists before it): a unit test writes every read type’s normalised DEFAULT_MASK and its valid path list to apps/bonsapi/src/grpc/field_selection/snapshots/<read_type>.txt and compares them with the committed file. Any change to a default, or to the set of valid paths, shows up as a diff in review, so it is always deliberate.
12. Observability and SLOs (0.1.11) #
Traces. The MCP sends traceparent on every call (src/grpc/shared/metadata.ts). bonsapi already continues it (extract_parent_trace_context), so one trace runs from the MCP tool call through bonsapi to the database. Each RPC span carries rpc.system=grpc, rpc.service, rpc.method, rpc.grpc.status_code, tofu.error.reason, tofu.read_mask.path_count, tofu.read_mask.is_default, tofu.load_groups, tofu.batch.size and tofu.response.bytes, and never a field value.
Metrics (OpenTelemetry, beside HttpMetrics in model/app/metrics.rs):
| Metric | Type | Attributes |
|---|---|---|
grpc.server.duration |
histogram, ms | service, method, code |
grpc.server.requests |
counter | service, method, code, reason |
grpc.server.response.size |
histogram, bytes | service, method |
grpc.read_mask.paths |
counter, one per requested path | service, method, path (from the generated PATHS list, unknown paths counted as _invalid, so cardinality is bounded) |
grpc.read_mask.default |
counter | service, method, is_default |
grpc.quota.rejections |
counter | service, kind (read or write) |
The path counter shows which fields callers actually use. It is the evidence for adding a path to a default, or for keeping one there.
SLOs, per service, over 28 days:
- Availability: 99.9% of calls do not end in
INTERNAL,UNAVAILABLEorDEADLINE_EXCEEDED. Client errors (INVALID_ARGUMENT,PERMISSION_DENIED,NOT_FOUND,ABORTED,RESOURCE_EXHAUSTED) do not count against it. The error budget is 0.1%, about 40 minutes of full outage in 28 days. - Latency: p95 under 300 ms for
GetandBatchGet, under 800 ms forListandUpdate.
Alerts (Datadog monitors on the metrics above, multi-window burn rate):
- Page: the budget burns at 14.4× over both 1 hour and 5 minutes.
- Ticket: the budget burns at 6× over both 6 hours and 30 minutes.
- Ticket: p95 latency is over its SLO for 30 minutes.
13. Worked examples #
Examples are in proto JSON form. Values are made up.
Direct expense: default read #
// GetDirectExpense
{ "id": "6f0c…", "entity_id": "a1b2…" } // no read_mask: default mask
// reply (about 1.3 KB for two lines)
{
"id": "6f0c…", "entity_id": "a1b2…", "etag": "W/\"9c41e0…\"",
"extraction_id": "3c4d…", "status": "READY", "publish_sync_status": "NOT_SYNCED",
"data": {
"id": "d9e1…", "action_type": "UPDATE",
"title": "Taxi", "transaction_date": 1794700800000, "currency_id": "JPY",
"total_amount": 2310, "line_amount_type": "INCLUSIVE", "contact_name": "Example Taxi",
"line_items": [
{ "id": "l1…", "description": "Fare", "quantity": 1, "unit_price": 2100, "amount": 2100, "tax_amount": 210 },
{ "id": "l2…", "description": "Toll", "quantity": 1, "unit_price": 210, "amount": 210, "tax_amount": 0 }
]
}
}
No metadata, no bboxes, no traces, no sync_activities: they are not in the default, so their queries do not run.
Invoice: batch get with an explicit mask and a heavy field #
// BatchGetInvoices
{
"entity_id": "a1b2…",
"ids": ["i1…", "i2…", "i3…"],
"read_mask": "data.invoice_number,data.total_amount,data.metadata.traces"
}
// reply: per-item results in request order
{
"results": [
{ "id": "i1…", "invoice": { "id": "i1…", "entity_id": "a1b2…", "etag": "…",
"data": { "id": "…", "action_type": "CREATE", "invoice_number": "INV-001", "total_amount": 11000,
"metadata": { "traces": { "ap_bill_data.invoice_number": { … } } } } } },
{ "id": "i2…", "error": { "code": 5, "message": "Not found.",
"details": [{ "@type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": "RESOURCE_NOT_FOUND", "domain": "bonsapi.gotofu.com",
"metadata": { "resource_type": "invoice" } }] } },
{ "id": "i3…", "invoice": { … } }
]
}
data.metadata.traces is heavy, so it comes back only because it was named. data.metadata.bboxes stays out.
Bank statement: header, then transactions by page #
// GetBankStatement: latest version with data, header only
{ "id": "b1…", "entity_id": "a1b2…", "version": { "latest": {} } }
// reply: default mask, data.transaction_count = 33, no transactions
// ListBankTransactions
{
"bank_statement_id": "b1…", "entity_id": "a1b2…",
"page_size": 50, "read_mask": "date,description,amount"
}
// reply
{ "transactions": [ { "id": "t1…", "date": 1794700800000, "description": "…", "amount": -1200 }, … ],
"next_page_token": "" }
version is a oneof { Latest latest; LatestVerified latest_verified; string data_id; }. latest skips a version that is still being written, so a re-read never returns an empty version (the details are in 0.5).
Extraction: link fields for pairing #
// ListExtractions
{
"entity_id": "a1b2…",
"filter": "type = \"DIRECT_EXPENSE\" AND status = \"NEEDS_REVIEW\"",
"order_by": "created_at desc",
"page_size": 100,
"read_mask": "id,status,document_id,direct_expense_id"
}
// reply (the caller holds DIRECT_EXPENSE:READ, so direct_expense_id is allowed)
{ "extractions": [ { "id": "e1…", "entity_id": "a1b2…", "etag": "…", "status": "NEEDS_REVIEW",
"document_id": "doc1…", "direct_expense_id": "6f0c…" }, … ],
"read_time": "2026-11-18T02:40:00Z",
"resume_time": "2026-11-18T02:35:00Z",
"next_page_token": "eyJr…" }
Document: unknown path #
// GetDocument
{ "id": "doc1…", "entity_id": "a1b2…", "read_mask": "file_name,private_metadata" }
// error
{
"code": 3, "message": "read_mask has 1 invalid path.",
"details": [
{ "@type": "type.googleapis.com/google.rpc.ErrorInfo", "reason": "INVALID_READ_MASK",
"domain": "bonsapi.gotofu.com",
"metadata": { "invalid_paths": "private_metadata",
"valid_paths": "id,entity_id,file_name,mime_type,size,status,source,…" } },
{ "@type": "type.googleapis.com/google.rpc.BadRequest",
"field_violations": [ { "field": "read_mask.paths[1]", "description": "Unknown path." } ] }
]
}
Direct expense: partial update with a stale etag #
// UpdateDirectExpenseData
{
"id": "6f0c…", "entity_id": "a1b2…",
"data": { "external_account_id": "acc-610", "line_items": [ { "id": "l2…", "external_account_id": "acc-620" } ] },
"update_mask": "data.external_account_id,data.line_items.external_account_id",
"etag": "W/\"9c41e0…\"",
"request_id": "0b7e…",
"read_mask": "data.external_account_id,data.line_items.external_account_id"
}
// error: someone saved in between
{
"code": 10, "message": "The direct expense changed since it was read.",
"details": [
{ "@type": "type.googleapis.com/google.rpc.ErrorInfo", "reason": "ETAG_MISMATCH",
"domain": "bonsapi.gotofu.com", "metadata": { "current_etag": "W/\"77ab02…\"" } },
{ "@type": "type.googleapis.com/bonsapi_grpc.direct_expense.v1.ConflictDetail",
"current_etag": "W/\"77ab02…\"",
"current": { "id": "6f0c…", "etag": "W/\"77ab02…\"", "data": { "external_account_id": "acc-600", … } } }
]
}
The caller merges with current and retries with the new etag and a new request_id. ETAG_MISMATCH is a stored outcome (§7), so resending the old id would replay this error.
14. Open questions for review #
For the MCP owner and the bonsapi owners (0.1.4). Approval is a comment on ENG-9842 from two engineers, including the MCP owner.
- Should the bank statement default include the first page of transactions, rather than none?
- Is a required
etagon updates acceptable for the MCP write tools? It means a read before every write, which the tools mostly do already. - The quota starting values (600 read and 120 write units per organisation per minute).
- Should
ListExtractionsandListDocumentsaccept a filter across entities for organisation-wide reads, or stay one entity per call?