E2E tests #

Classification: Restricted (describes source code, infrastructure, and access control).

End-to-end tests for tofu-external-mcp with the real Clerk OAuth app and real bonsapi. The scenarios, scripts and coverage map live in tests/e2e/tofu-external-mcp/. Claude Code runs the tofu-external-mcp-e2e skill (.claude/skills/tofu-external-mcp-e2e/) against them and reports PASS/FAIL per scenario.

CI workflow #

.github/workflows/tofu-mcp-e2e.yaml runs four jobs against dev. It deploys nothing: the branch’s own scenarios and scripts run, but against whatever dev currently serves (dev tracks main), so a fix in the MCP itself is only exercised after it merges.

The one input, ai_extraction (default on), also runs documents-ai, which uses dev AI quota. Each Claude job fails unless its e2e-result.json shows every requested scenario as PASS and blocked as false. A scenario reported SKIPPED is a warning, and a job where every scenario was skipped fails. The report goes to the run summary and is uploaded as an artifact (tofu-mcp-e2e-read-report, tofu-mcp-e2e-write-report).

Local run #

scripts/run-local.sh (the tofu-external-mcp-e2e mise task) needs the local stack from mise run dev, doppler, claude and MCP_E2E_EXPECTED_ORG_ID. It logs in against the local server’s own /authorize with the loopback redirect http://localhost:28765/callback that .mcp.json pins, runs the auth checks, then runs the scenarios (default smoke,read,negative; pass none to stop after the checks) with the Claude CLI, and prints the report. Secrets come from Doppler bonsai / dev_local, and the credentials live in a temporary directory that is removed on exit.

Scenarios #

Coverage map #

Each scenario ends with a fenced tools block naming the tools it really calls. coverage.json lists the tools that are deliberately skipped, each with a reason (integration publish/import and submitter tools need a provider sandbox, update_organization_settings would change a shared org, the currency write tools need multi-currency enabled and no tool enables it, list_coupons and list_pricing_plans are served only to a sales agent, a few cannot be seeded through the MCP). A second block, tools-conditional, lists tools that run only when the dev org has the data or members they need (members, integrations, an org external id) and are otherwise SKIPPED; they count as accounted for but not as proven. The coverage job fails when a catalog tool is in none of the three places, when a tool is both skipped and exercised, or when a name is not in the catalog, so adding a tool to the catalog without a scenario or a skip entry fails CI (it also runs in the normal unit tests). The Claude runs deny every skipped tool at the tool level (--disallowedTools), so a skipped tool cannot be called by mistake or by injected instructions.

How authentication works #

The server accepts only Clerk OAuth access tokens (typ: at+jwt) whose client_id is its own OAuth app (Auth). Claude Code cannot complete an OAuth login in a headless run, so each run first logs in as the e2e test user itself: scripts/oauth-token.mjs runs the authorization-code + PKCE flow in headless Chromium (Playwright) with the server’s own OAuth app and writes MCP_E2E_CREDS_FILE outside the workspace. Details that matter:

• Which /authorize. The local server’s /authorize accepts the loopback redirect http://localhost:28765/callback, which is registered on the OAuth app. The deployed server’s /authorize rejects loopback addresses, so against dev the script authorizes straight at Clerk (<issuer>/oauth/authorize) with the same registered redirect. In both cases the redirect is never followed: the code is read off the 303 Location header and exchanged at the server’s /token.
• Cloudflare Access. The sign-in page is the webapp’s, behind Cloudflare Access, and browsers drop injected headers on redirect hops. When the redirect lands on the Cloudflare login page, the script makes one direct request to that host with the service-token headers to earn the CF_Authorization cookie, then starts the flow again.
• Organization. Clerk’s active organization belongs to the browser session, and the consent screen has no org picker. The script calls Clerk.setActive for the [DO NOT TOUCH] E2E ORG membership on the consent page, and requests the user:org:read scope, which is what puts org_id in the token. It refuses a token whose org_id is not MCP_E2E_EXPECTED_ORG_ID.

scripts/headers-helper.sh, a Claude Code headersHelper, serves that access token as the Authorization header, so Claude Code does not attempt OAuth itself. The helper does not refresh: the access token lives about 24 hours, longer than a run, and the credentials file holds no refresh token and no client secret (only the access token, its expiry, the client id and the Cloudflare service token).

How results are checked #

Claude reports on itself, so scripts/verify-trace.mjs cross-checks e2e-result.json against the run’s own tool-call trace. Each scenario reported PASS must have called at least 60% of the tools unique to it among the requested scenarios (its whole tools block when it shares every tool with another scenario). A scenario can also require evidence in a trace-markers block, for example negative must have seen error results from get_entity, delete_entity and create_entity. A missing trace, a scenario without a tools block, or an unknown scenario fails the job. Claude reads every scenario file up front, so the trace cannot say which scenario a call belonged to and calls are not attributed by position: a scenario that shares all its tools with another one (for example smoke with read) is only weakly checked. The check catches a PASS reported for a scenario that was not run, not a scenario that was run badly.

The Claude step has no Bash tool, only Skill, Read of the skill, scenario and fixture files, Edit of the two report files and the allowed MCP tools. The credentials file is outside the workspace, so Claude cannot read it.

Configuration #

Secrets are read from Doppler (bonsai / dev in CI, dev_local locally) with doppler run --only-secrets, so the login step sees only these:

GitHub:

No org id is configured: the login finds the [DO NOT TOUCH] E2E ORG membership by name and the run fails if the token is for another org. Set MCP_E2E_EXPECTED_ORG_ID in the environment to pin a specific Clerk org id as well.

The test user must be able to sign in with email and password alone (no extra verification step) and be a member of [DO NOT TOUCH] E2E ORG (override the name with E2E_ORG_NAME). Anyone who can dispatch the workflow on a branch can change the scripts that handle these secrets, so restrict the dev environment (required reviewers, allowed branches) in the repository settings.

Limits #

• CI tests whatever dev serves. Runs are serialised (tofu-mcp-e2e concurrency group).
• Writes are confined to the e2e-* entities the run creates, and the lifecycle scenario deletes e2e- entities older than 2 hours left by crashed runs. There is no document delete tool, so uploaded documents stay in storage after their entity is removed.
• If the dev org is still onboarding or at its entity limit, write scenarios report SKIPPED.
• The read and write jobs use the same user’s 60 tool calls per minute. Claude has no shell to wait out a rate-limit error, so run together they failed each other; the write job therefore starts after the read job.
• On the deployed dev hosts /.well-known/oauth-protected-resource and /.well-known/oauth-authorization-server currently answer a 301 loop (an ALB redirect behind a Cloudflare rule), so the two discovery checks fail there while passing locally. That is an infrastructure fault, not a test fault.
• Not covered: a token from a different OAuth client (403) or a genuinely expired signed token, since the suite cannot mint either (forged and unsigned variants are checked); audit-log and trace correlation (no log access from CI, see Audit logging); integrations and publish flows.
• Claude can call every tool that is not skipped, including delete_entity, on any entity in the e2e org. The skill limits it to entities whose slug matches ^e2e-([a-z]+-)?[0-9]{10}$ and tells it to ignore instructions found in tool output, but nothing enforces that technically, so keep valuable data out of the e2e org.
• Claude drives the sweeps, so results can vary; reproduce a failure from the report before treating it as a regression.