Chat evals in CI

Chat evals in CI #

Classification: Restricted (describes source code, infrastructure, and access control).

.github/workflows/chat-evals.yml runs the assistant’s smoke evals on pull requests that touch the chat prompts or harness, and the Chat evals check fails the merge when a case fails. The cases, the runner and the full suite are described in apps/tofu-web/evals/README.md.

Which pull requests run it #

A non-draft pull request from a branch in this repository runs the smoke set when it changes any of:

Path What it is
apps/tofu-web/src/server/ai/** The orchestrator, judge, roles, limits and MCP scoping
libs/typescript/ai/prompts_encrypted/chat-agents/**, guardrails/** The persona and guardrail prompts
libs/typescript/chat-orchestrator/src/** The shared orchestration code
apps/tofu-web/evals/**, apps/tofu-web/scripts/chat-eval.ts, the workflow itself The cases and the gate

Test files (__tests/, *.test.ts) do not count. Anything else runs nothing and costs nothing; the check still reports, as passed. It runs on open, ready-for-review and each push, and a newer commit cancels the run in progress. In the merge queue the check passes without running: the evals ran on the pull request.

Label Effect
run-evals Runs the smoke set whatever the paths, for example when a command or tool description under apps/tofu-web/src/commands changed
skip-evals Skips the run and passes the check. For a maintainer, when the eval budget is exhausted

Adding either label starts a run. Any other label is ignored: its run cancels nothing and reports under another name (Chat evals / Label ignored), so it cannot replace a failed Chat evals with a skipped one. Removing a label takes effect on the next push or re-run.

Pull requests from forks and from Dependabot never run the evals: they get neither the secrets nor the OIDC trust. A maintainer pushes the change to a branch here first.

What a run does #

  1. --check validates the dataset, that the smoke set is estimated within 2 USD, and that every catalog model has a price.
  2. Logs in as the eval user (the tofu MCP login action) and runs --tag smoke --models primary --no-grade against the dev stack, with Redis as a service container. Every write is declined, as in a local run. A case that errors is run once more before it counts.
  3. Writes the result to the job summary and one pull request comment: counts, cost and the ids of the cases that failed, errored or were not gated. Never transcripts or failure text, which can hold dev customer data; the job log has the failed checks and, for each case that did not pass, an outline with no words in it: the routing decision, the tools called by name, the approvals declined and the reply’s length.
Result Meaning The check
passed Every gated smoke case passed passes
failed A case failed its checks fails
incomplete The run stopped at the 2 USD ceiling, a case errored twice, a model had no price, or no case was gated fails
eval budget exhausted The workspace’s spend limit stopped the calls, and no case had failed. Not a behaviour failure fails, until skip-evals

Models #

CI uses the chat’s own models with no override and no API key: each provider trusts the job’s GitHub OIDC token (LLM_IDENTITY_SOURCE=github-actions). Anthropic is required. OpenAI is optional (added 2026-11-17): the fast tier has no Anthropic model (GPT-5.4 mini, then Gemini 3 Flash), so without OpenAI’s ids the run passes --anthropic-only, and a smoke case the judge routes to that tier is skipped and listed as not gated, and the route and fact fallbacks are unavailable. Gemini has no identity in CI (decided 2026-11-17): it is only ever a fallback, which a primary run never calls, and Google Cloud budgets alert without stopping spend. Routing is judged by System One, which is billed separately and only counted.

Budget #

Limit Where it is enforced
2 USD per run --max-cost 2 in the runner. It does not start a case expected to cross the ceiling, and inside a case it refuses every further model leg once the ceiling is reached, so a run overshoots by at most the leg in flight
Per month No cap in the runner: each provider’s calls are held to the spend limit of the workspace or project its service account belongs to. A 100 USD monthly cap is that limit, on a workspace kept for evals

What a pull request can reach #

A pull request can change the workflow and the code it runs, so nothing in the workflow is relied on to hide a secret from its author. The limit is what the chat-evals environment holds:

Credential Scope
DOPPLER_TOKEN A read-only service token for the dev Doppler config (decided 2026-11-17), which holds dev model keys. Any workflow on a branch here can already read that config through the unprotected dev environment, so this adds no reach
GPG_SECRET_KEY_AI_LIB The repository secret every CI job already decrypts the prompts with
Claude and OpenAI OIDC Each provider’s rule accepts only this repository’s chat-evals environment, so a pull request can spend from the workspace or project that rule’s service account belongs to, inside its spend limit

The run drops every model key variable from the process and reads only the secrets --only-secrets names, so a dev model key never pays for a run.

Setup #

One-time, outside the repository:

Where What
Anthropic console A federation rule for GitHub’s OIDC issuer (https://token.actions.githubusercontent.com), audience https://api.anthropic.com, subject repo:<owner>/<repo>:environment:chat-evals, on a service account in the workspace the evals spend from. The deployed services’ rule trusts an AWS role, which a runner does not hold
OpenAI (optional) An identity provider for the same issuer and subject, audience https://api.openai.com/v1, mapped to a service account in the project the evals spend from
Doppler In the dev config of bonsai: CHAT_EVALS_ANTHROPIC_FEDERATION_RULE_ID, CHAT_EVALS_ANTHROPIC_ORGANIZATION_ID, CHAT_EVALS_ANTHROPIC_SERVICE_ACCOUNT_ID, CHAT_EVALS_ANTHROPIC_WORKSPACE_ID; optionally CHAT_EVALS_OPENAI_IDENTITY_PROVIDER_ID and CHAT_EVALS_OPENAI_SERVICE_ACCOUNT_ID (identifiers, not secrets; prefixed because the deployed services read the unprefixed names); CHAT_EVALS_E2E_USER_EMAIL, CHAT_EVALS_E2E_USER_PASSWORD and CHAT_EVALS_E2E_USER_TOTP_SECRET (the authenticator seed, base32), for a user who signs in with email, password and an authenticator code and belongs to exactly one organisation; CHAT_EVALS_VARS, space-separated name=value pairs for the smoke cases’ requires, from that organisation: entity=… extraction=…. Everything else the run reads is already there. A read-only service token for that config
GitHub environment chat-evals The secret DOPPLER_TOKEN (the token above). No required reviewers, or every run waits for one
Branch protection Require Chat evals

The workspace id is required: without it the calls would not be held to one workspace’s limit. The evals log in as a user of their own, not the MCP e2e suite’s, so the two do not share one rate limit (60 tool calls a minute per user); runs on several pull requests at once still share it, which is what the retry of an errored case is for.

Changing things #

To Do
Run the full suite mise run tofu-web-eval locally (about 6 to 8 USD per model)
Run what CI runs mise run tofu-web-eval -- --tag smoke --models primary --no-grade --max-cost 2 --var entity=… --var extraction=…
Add a case Add it to a file under apps/tofu-web/evals/cases/, then mise run tofu-web-eval -- --check
Add a smoke case Add the smoke tag to a hard case. --check refuses a soft one, and a set estimated over 2 USD (0.10 USD per turn, so 20 turns)
Change the pass threshold The gate is 100% of the smoke set. To let a case stop gating, remove its smoke tag
Change the per-run ceiling MAX_COST_USD in the workflow and SMOKE_MAX_USD in scripts/chat-eval.ts, together
Correct the per-turn estimate TURN_ESTIMATE_USD in scripts/chat-eval.ts. A run prints a line when its measured mean turn is above it
Change the trigger paths The pattern in the workflow’s gate job, and the table above