Chat evals in CI #
Classification: Restricted (describes source code, infrastructure, and access control).
.github/workflows/chat-evals.yml runs the assistant’s smoke evals on pull requests that touch the chat prompts or harness, and the Chat evals check fails the merge when a case fails. The cases, the runner and the full suite are described in apps/tofu-web/evals/README.md.
Which pull requests run it #
A non-draft pull request from a branch in this repository runs the smoke set when it changes any of:
| Path | What it is |
|---|---|
apps/tofu-web/src/server/ai/** |
The orchestrator, judge, roles, limits and MCP scoping |
libs/typescript/ai/prompts_encrypted/chat-agents/**, guardrails/** |
The persona and guardrail prompts |
libs/typescript/chat-orchestrator/src/** |
The shared orchestration code |
apps/tofu-web/evals/**, apps/tofu-web/scripts/chat-eval.ts, the workflow itself |
The cases and the gate |
Test files (__tests/, *.test.ts) do not count. Anything else runs nothing and costs nothing; the check still reports, as passed. It runs on open, ready-for-review and each push, and a newer commit cancels the run in progress. In the merge queue the check passes without running: the evals ran on the pull request.
| Label | Effect |
|---|---|
run-evals |
Runs the smoke set whatever the paths, for example when a command or tool description under apps/tofu-web/src/commands changed |
skip-evals |
Skips the run and passes the check. For a maintainer, when the eval budget is exhausted |
Adding either label starts a run. Any other label is ignored: its run cancels nothing and reports under another name (Chat evals / Label ignored), so it cannot replace a failed Chat evals with a skipped one. Removing a label takes effect on the next push or re-run.
Pull requests from forks and from Dependabot never run the evals: they get neither the secrets nor the OIDC trust. A maintainer pushes the change to a branch here first.
What a run does #
--checkvalidates the dataset, that the smoke set is estimated within 2 USD, and that every catalog model has a price.- Logs in as the eval user (the tofu MCP login action) and runs
--tag smoke --models primary --no-gradeagainst the dev stack, with Redis as a service container. Every write is declined, as in a local run. A case that errors is run once more before it counts. - Writes the result to the job summary and one pull request comment: counts, cost and the ids of the cases that failed, errored or were not gated. Never transcripts or failure text, which can hold dev customer data; the job log has the failed checks and, for each case that did not pass, an outline with no words in it: the routing decision, the tools called by name, the approvals declined and the reply’s length.
| Result | Meaning | The check |
|---|---|---|
| passed | Every gated smoke case passed | passes |
| failed | A case failed its checks | fails |
| incomplete | The run stopped at the 2 USD ceiling, a case errored twice, a model had no price, or no case was gated | fails |
| eval budget exhausted | The workspace’s spend limit stopped the calls, and no case had failed. Not a behaviour failure | fails, until skip-evals |
Models #
CI uses the chat’s own models with no override and no API key: each provider trusts the job’s GitHub OIDC token (LLM_IDENTITY_SOURCE=github-actions). Anthropic is required. OpenAI is optional (added 2026-11-17): the fast tier has no Anthropic model (GPT-5.4 mini, then Gemini 3 Flash), so without OpenAI’s ids the run passes --anthropic-only, and a smoke case the judge routes to that tier is skipped and listed as not gated, and the route and fact fallbacks are unavailable. Gemini has no identity in CI (decided 2026-11-17): it is only ever a fallback, which a primary run never calls, and Google Cloud budgets alert without stopping spend. Routing is judged by System One, which is billed separately and only counted.
Budget #
| Limit | Where it is enforced |
|---|---|
| 2 USD per run | --max-cost 2 in the runner. It does not start a case expected to cross the ceiling, and inside a case it refuses every further model leg once the ceiling is reached, so a run overshoots by at most the leg in flight |
| Per month | No cap in the runner: each provider’s calls are held to the spend limit of the workspace or project its service account belongs to. A 100 USD monthly cap is that limit, on a workspace kept for evals |
What a pull request can reach #
A pull request can change the workflow and the code it runs, so nothing in the workflow is relied on to hide a secret from its author. The limit is what the chat-evals environment holds:
| Credential | Scope |
|---|---|
DOPPLER_TOKEN |
A read-only service token for the dev Doppler config (decided 2026-11-17), which holds dev model keys. Any workflow on a branch here can already read that config through the unprotected dev environment, so this adds no reach |
GPG_SECRET_KEY_AI_LIB |
The repository secret every CI job already decrypts the prompts with |
| Claude and OpenAI OIDC | Each provider’s rule accepts only this repository’s chat-evals environment, so a pull request can spend from the workspace or project that rule’s service account belongs to, inside its spend limit |
The run drops every model key variable from the process and reads only the secrets --only-secrets names, so a dev model key never pays for a run.
Setup #
One-time, outside the repository:
| Where | What |
|---|---|
| Anthropic console | A federation rule for GitHub’s OIDC issuer (https://token.actions.githubusercontent.com), audience https://api.anthropic.com, subject repo:<owner>/<repo>:environment:chat-evals, on a service account in the workspace the evals spend from. The deployed services’ rule trusts an AWS role, which a runner does not hold |
| OpenAI (optional) | An identity provider for the same issuer and subject, audience https://api.openai.com/v1, mapped to a service account in the project the evals spend from |
| Doppler | In the dev config of bonsai: CHAT_EVALS_ANTHROPIC_FEDERATION_RULE_ID, CHAT_EVALS_ANTHROPIC_ORGANIZATION_ID, CHAT_EVALS_ANTHROPIC_SERVICE_ACCOUNT_ID, CHAT_EVALS_ANTHROPIC_WORKSPACE_ID; optionally CHAT_EVALS_OPENAI_IDENTITY_PROVIDER_ID and CHAT_EVALS_OPENAI_SERVICE_ACCOUNT_ID (identifiers, not secrets; prefixed because the deployed services read the unprefixed names); CHAT_EVALS_E2E_USER_EMAIL, CHAT_EVALS_E2E_USER_PASSWORD and CHAT_EVALS_E2E_USER_TOTP_SECRET (the authenticator seed, base32), for a user who signs in with email, password and an authenticator code and belongs to exactly one organisation; CHAT_EVALS_VARS, space-separated name=value pairs for the smoke cases’ requires, from that organisation: entity=… extraction=…. Everything else the run reads is already there. A read-only service token for that config |
GitHub environment chat-evals |
The secret DOPPLER_TOKEN (the token above). No required reviewers, or every run waits for one |
| Branch protection | Require Chat evals |
The workspace id is required: without it the calls would not be held to one workspace’s limit. The evals log in as a user of their own, not the MCP e2e suite’s, so the two do not share one rate limit (60 tool calls a minute per user); runs on several pull requests at once still share it, which is what the retry of an errored case is for.
Changing things #
| To | Do |
|---|---|
| Run the full suite | mise run tofu-web-eval locally (about 6 to 8 USD per model) |
| Run what CI runs | mise run tofu-web-eval -- --tag smoke --models primary --no-grade --max-cost 2 --var entity=… --var extraction=… |
| Add a case | Add it to a file under apps/tofu-web/evals/cases/, then mise run tofu-web-eval -- --check |
| Add a smoke case | Add the smoke tag to a hard case. --check refuses a soft one, and a set estimated over 2 USD (0.10 USD per turn, so 20 turns) |
| Change the pass threshold | The gate is 100% of the smoke set. To let a case stop gating, remove its smoke tag |
| Change the per-run ceiling | MAX_COST_USD in the workflow and SMOKE_MAX_USD in scripts/chat-eval.ts, together |
| Correct the per-turn estimate | TURN_ESTIMATE_USD in scripts/chat-eval.ts. A run prints a line when its measured mean turn is above it |
| Change the trigger paths | The pattern in the workflow’s gate job, and the table above |