Rate Limiting

Rate Limiting #

Classification: Restricted (describes source code, infrastructure, and access control).

tofu-external-mcp enforces 60 tools/call per rolling minute, per Clerk user. The count is shared across pods, so it is one budget per person however many connections they hold.

Why it exists #

The threat is not a human clicking too fast. It is agent amplification: one prompt turning into an unbounded number of tool calls.

Shape Example Caught?
Runaway agent loop “List every entity, and for each list all its documents and extractions” — a nested loop the model runs to exhaustion Yes
Prompt-injection enumeration A document’s free-text field carries an instruction to enumerate everything; the user innocently asks Claude to summarise it Yes — one session doing legitimate-looking work, which a per-session budget would miss
Deliberate custom client Someone writes their own MCP client and batches or churns sessions to go faster Yes, for tool calls — see the session-creation gap below
Unauthenticated flood Raw connection-level abuse with no valid token No — rejected at clerkAuth before the limiter, so only an edge/WAF control would see it

One thing it does not meter is session creation. initialize is free — deliberately, so protocol traffic does not eat a caller’s tool-call budget — while the session store is an in-memory map evicted only on a 30-minute idle timeout. Churning sessions buys no extra tool-call budget, because the budget is keyed to the user rather than the session, but the allocation itself is uncapped: an allowlisted caller can create sessions faster than they expire. That is a property of the session store rather than of this limiter, and is worth a look when cross-pod session state is picked up.

Sustained breaches are the signal ENG-7046’s suspension kill-switch acts on. This limiter bounds abuse and makes it visible; the suspension is what stops a determined actor.

How it works #

What counts #

tools/call messages, not HTTP requests. One POST usually carries one tool call, but the transport also accepts a JSON-RPC batch array and executes every message in it — so counting requests could be evaded by putting N calls in one body.

Everything else is free: initialize, tools/list, ping, notifications, the long-lived SSE stream and the session teardown. “60/min” therefore means 60 actual tool calls, not 60 units of mixed protocol traffic.

Budget is spent before the transport validates the request, so a misconfigured client (wrong Accept header, stale session id) still pays for calls that never reach a tool. That is deliberate — charging only for valid requests would hand an abuser an unlimited supply of cheap invalid ones.

Who it counts against #

The sub claim of the verified JWT — the caller’s Clerk user id. The limiter runs after authentication, so the identity cannot be forged.

Deliberately not scoped by organisation: one user is one budget however many orgs they act in or sessions they run in parallel. A token with no sub falls back to the org id; a request carrying neither is let through, but logged, so the gap is visible rather than silent.

The window #

A sliding 60 seconds, not a fixed bucket: the window is measured backwards from each request, so calls age out continuously rather than everyone’s count resetting at the top of the minute. A batch is admitted or refused as a unit — 40 calls arriving with 30 of the budget left is rejected whole, not partially applied.

What a refused call gets back #

  • 429 for a caller over the limit, with Retry-After set to when their budget genuinely frees up (not a flat minute) and RateLimit-Remaining set to what they actually have left — a batch can be refused while smaller calls would still fit.
  • 400 for a batch larger than the whole limit. That can never be admitted however long the client waits, so a 429 would be an instruction to retry forever; the message names both numbers and says to split the batch.

Both are JSON-RPC error envelopes, matching how the transport reports its own rejections.

When Redis is unavailable #

It fails open — calls proceed unlimited. A rate limiter that turns a Redis outage into a service outage is worse than the abuse it prevents.

That window is audited rather than swallowed, and a repeated failure stops re-trying for a few seconds so an outage costs one timeout instead of one per request. If REDIS_HOST is unset entirely, the service logs service_degraded once at startup — otherwise a missing secret would start a healthy-looking pod with the limiter quietly switched off.

Design decisions #

Recorded so the next person does not re-litigate them.

Not AWS WAF #

The obvious first move was to copy bonsapi, which rate-limits in AWS WAF keyed on the Tofu-API-Key header — that works because the API key is a stable value the client sends verbatim, so the header value is the identity. It does not transfer:

  • WAF cannot read a JWT claim. Rate-based custom keys cover IP, headers, cookies, query args, method, path and TLS fingerprints. There is no JWT or claim key type, and no way to select a field out of one.
  • Keying on the raw Authorization value is broken. Clerk session tokens live 60 seconds, so the key would change every minute and a counter would never accumulate. OAuth tokens live longer but are per-credential: the budget resets on refresh and every device gets its own.
  • IP keying is near-useless here. Remote MCP traffic from claude.ai arrives from Anthropic’s egress IPs, so every user would collapse into one key.
  • Making WAF key on a user id would need a CloudFront layer decoding the JWT and injecting a header for the ALB’s WAF to key on — a new distribution, a DNS cutover, origin lockdown so the header cannot be spoofed, and a rate key that CloudFront cannot verify the signature of. Judged overkill.

Doing it in the application puts the limiter after authentication, where the user id is already known and trustworthy.

Not an in-process counter #

Prod runs two replicas with no load-balancer stickiness, so a per-pod counter would permit 2 × 60 = 120/min — and would silently change meaning the moment anyone edited the replica count or added an HPA. Dividing the limit by replica count is worse: it breaks on any scaling change, and uneven distribution makes it simultaneously too strict for some users and too loose for others.

Not a fixed window #

A fixed per-minute bucket is cheaper, but a client straddling a boundary can land 120 calls in a 60-second straddle. The sliding window also makes the response headers honest — real remaining budget and a real retry time, rather than the hardcoded zeros the bonsapi WAF rule emits.

Keying alternatives rejected #

Option Why not
Session id An abusive client re-initializes for a fresh budget. Also misses prompt-injection enumeration, which is one legitimate session.
org_id alone One noisy user would consume the entire org’s budget.
client_id Useless as a key — claude.ai is a single client id shared by all our users. Kept as a metric label so Datadog can distinguish it from a custom client.

Deferred, not dropped #

  • A per-org ceiling on top of the per-user limit. The gap is an org adding N users for N × 60 — but reaching that needs the org to already hold is_external_mcp_enabled, an allowlist we control. Adding it later is additive, and keys are namespaced from the start to accommodate it.
  • Per-tool cost weighting — a list_documents that pages 10k rows costing more than a get_entity. The right answer if the problem turns out to be expensive tools rather than many calls, but the weights would be guesswork without usage data.

Operating it #

Is the limit right? #

60/min is not yet evidence-based. It matches what bonsapi already enforces; it has not been checked against observed agent fan-out. The number to compare against is the peak tool-calls-per-minute a real Claude session naturally produces — see the benchmarking protocol in ENG-7045.

To change it, set TOFU_EXTERNAL_MCP_RATE_LIMIT in deployment/resources/tofu-external-mcp/deployment.yaml — a deploy, not a release.

What to watch #

  • Rejections — the rate_limited audit event, grouped by actor_user_id. One user spiking is a runaway loop or an injection; many users at once suggests the limit is too low. Split by http_status: 429 means back off (or suspend via ENG-7046), 400 means the client should split its batch. Different problems — don’t monitor them as one number.
  • Fail-open windows — rate_limited with failed_open:true. While these fire there is no limit in effect, so this wants its own monitor. Count data.suppressed_since_last, not lines — the event is damped so a sustained outage does not emit one line per request.
  • service_degraded with reason:redis_not_configured — emitted once at startup if REDIS_HOST is unset, which means the limiter is not running at all.
  • Tool-call volume — tool_call_request already carries actor_user_id, actor_org_id, actor_client_id and tool_name, so per-user call rate is a log-based metric over events this service already emits. See Audit Logging & Trace Correlation.

Running the tests #

The suites covering this run against a real Redis, because the properties worth proving — that concurrent calls cannot both slip past the check, and that a batch is admitted or refused as a unit — are properties of Redis, not of the code around it. docker compose up -d redis, then mise run tofu-external-mcp-test.

They skip when Redis is unreachable rather than failing, so a checkout that has not run mise run dev is not blocked — a [skip] line says so. The skip cannot hide in CI: the TypeScript job declares a Redis service container, and the gate hard-fails instead of skipping when CI is set, so deleting that service breaks the build rather than quietly dropping the suites.