Triggering Incidents #

Every way an incident can start, and how each source reaches the owning team.

Classification: Internal-Only

When to Use This Page #

• You want to know why you were paged
• You are adding a new monitor that should page
• You spotted a problem and need to raise it

Trigger Sources #

P1 Monitors: End to End #

A P1 monitor always triggers an incident. No human decision is needed.

Phase 1: Monitor Fires #

Goal: Detect customer-facing impact without waiting for a human

The P1 monitor owned by AI, Frontend, Backend or Infra goes into alert.

Phase 2: Incident Declared #

Goal: Put the incident on record immediately

1. Datadog auto-declares an incident from the monitor
2. Severity is set to SEV-1 or SEV-2 per the monitor
3. Ken Kanai is set as Commander
4. The owning team is set as the responding team

Phase 3: Team Paged #

Goal: Get the owning team responding

1. One owning-team member is paged by round robin (@oncall-<team>)
2. The Admin team is notified
3. Escalation follows the escalation policy

Phase 4: Slack Channel Created #

Goal: One place for all incident discussion

1. Datadog creates #incident-<incident-id>-<date>, for example #incident-142-20261005
2. The owning team’s on-call members are added to the channel
3. Ken Kanai is added to the channel
4. The incident link and monitor link are posted as the first message

Configuring Monitors #

Creating and Approving Monitors #

Each team creates its own monitors. The Admin team approves the priority.

Step 1: Team Creates the Monitor #

1. The owning team creates the monitor in Datadog
2. Tag it with team:<ai|frontend|backend|infra> and service:<name>
3. Propose a priority in the monitor description, for example Proposed: P1
4. Leave the paging handles out until the priority is approved

Step 2: Admin Team Approves the Priority #

1. Ask the Admin team in Slack to review the monitor
2. Share the monitor link and the proposed priority
3. Admin confirms or changes the priority
4. Admin sets the final priority on the monitor

P1 and P2 monitors always need Admin approval. Do not set either priority yourself.

Step 3: Team Adds the Paging Handles #

Once approved, add the handles that match the priority. See the message templates below.

Who Owns Which Monitors #

P1 Monitor Message #

A P1 monitor must both declare the incident and page the team.

{{#is_alert}}
@incident-critical @oncall-infra

## WHATS HAPPENING
Container terminated state in kube_cluster_name bonsai-app-eks-cluster-prod
for container bonsapi is greater than {{threshold}} over the last 5 minutes.

## IMPACT
<who and what is affected>

## Related links
- Logs
- Metrics Explorer (kube_pod_container_status_terminated)
{{/is_alert}}

{{#is_warning}}
@oncall-infra
Terminated containers rising for bonsapi in bonsai-app-eks-cluster-prod.
{{/is_warning}}

{{#is_recovery}}
@oncall-infra
Recovered.
{{/is_recovery}}

The screenshot below shows the older layout with handles on the first line. Move them inside the blocks as above.

P2-P3 Monitor Message #

P2 and P3 monitors page the on-call but do not create an incident. Declare one manually if impact is confirmed.

{{#is_alert}}
@oncall-backend

## WHATS HAPPENING
Worker DLQ depth rising on {{service.name}}.

## IMPACT
<who and what is affected>

## Related links
- Logs
{{/is_alert}}

{{#is_warning}}
@oncall-backend
Worker DLQ depth approaching threshold on {{service.name}}.
{{/is_warning}}

Rules for all monitors:

• Each monitor has exactly one owning team.
• Tag it with team:<ai|frontend|backend|infra> and service:<name>.
• P1 and P2 priorities are set only by the Admin team.
• Only P1 monitors use @incident-critical.
• Every message has WHATS HAPPENING, IMPACT and Related links sections.
• @incident-critical goes only inside {{#is_alert}}.
• @oncall-<team> goes inside {{#is_alert}} and {{#is_warning}}.
• @oncall-<team> inside {{#is_recovery}} is optional.

Monitor Priority #

P4-P5 Slack Alerts #

P4 and P5 monitors never page. They post to a Slack channel by environment.

• Put the handle inside {{#is_alert}} and {{#is_warning}}.
• Use the handle that matches the monitor’s environment.
• Copy the handle exactly; Slack handles are set up with that casing.

Prod Deploy Failure #

The notify-failures job in .github/workflows/prod-deploy.yaml fires when any prod deploy job fails. It still uses the PagerDuty action and is being moved to Datadog On-Call under the Infra team.

When it fires:

1. Open the failed run in GitHub Actions
2. Check the release thread in Slack for the tag and notes
3. Follow Deployment Monitoring
4. Declare an incident only if production is impacted

Manual Triggers #

From Datadog #

1. Open Datadog → Incidents → Declare Incident
2. Set Severity to SEV-3 and Commander to Ken Kanai
3. Optionally attach the triggering monitor or dashboard

Severity Review #

Manual incidents always start at SEV-3, even when raised from a monitor.

1. Post in the incident channel why it may be more severe
2. Invite the Admin team to the channel to review the impact
3. If Admin agrees, Ken escalates it to SEV-2 or SEV-1

When to Declare Manually #

P1 monitor fired?                 → Already declared, join the channel
Customers cannot use a core flow? → Declare
Data is wrong, lost or exposed?   → Declare
Security concern?                 → Declare (see Security below)
One user, workaround exists?      → Linear ticket, no incident
Unsure?                           → Declare. Downgrading is cheap.

Security Incidents #

• Declare with a vague title, for example Security investigation.
• Mark the incident private in Datadog.
• Do not discuss details in public channels.
• Page @oncall-admin and tell Ken Kanai immediately.

See Also #

• On-Call - Who receives the page
• Managing Incidents - What happens after declaring
• Monitoring & Alerting - Building and reading monitors