Managing Incidents

Managing Incidents #

How an incident is run in Datadog from declaration to resolution. Each incident gets its own Slack channel, and Ken Kanai is always the Incident Commander.

Classification: Internal-Only

When to Use This Page #

  • You just declared or acknowledged an incident
  • You are the lead responder for your team
  • You need the status update templates

Severity Levels #

Level Definition Response Time Update Cadence Examples
SEV-1 Complete outage or security breach Immediate 15 min API down, database unavailable
SEV-2 Major flow impaired 15 minutes 30 min Login failing, extraction stopped
SEV-3 Partial impact, no workaround 1 hour 60 min One integration sync failing

Only SEV-1 to SEV-3 are used. Minor issues with a workaround become Linear tickets, not incidents.

Severity of Manual Incidents #

  • Every manually declared incident starts at SEV-3, including ones raised from a monitor page.
  • Invite the Admin team to the incident channel to review it.
  • If Admin agrees, Ken escalates it to SEV-2 or SEV-1.

Roles #

Role Who Responsibility Datadog field
Incident Commander (IC) Ken Kanai, always Only person who can change the incident Commander
Lead Responder First to acknowledge from the owning team Drives investigation and mitigation Responders
Responder(s) Owning team on-call, other teams as pulled in Investigate and mitigate Responders
Admin team Admin on-call Oversight, cross-team coordination, escalation Responders
Communications Assigned by the IC Updates support, customers and management Responders
  • The Commander field is never changed from Ken Kanai.
  • Only Ken changes the incident: severity, state, fields, roles and timeline edits.
  • Everyone else asks Ken in the incident channel. Do not edit the incident yourself.
  • The Lead Responder speaks for the owning team in the channel.
  • Pull in another team by paging their handle, for example @oncall-infra.

Handing Off Lead Responder # 

🔁 LEAD RESPONDER HANDOFF
From: @current-lead
To: @new-lead
Current status: Mitigation applied, monitoring
Open threads: Confirming queue drain

Incident Lifecycle #

Phase 1: Declare #

Goal: Get the incident on record and the right people in the room

Step 1: Create the Incident #

P1 monitors declare the incident automatically. Otherwise declare from Slack or Datadog with Ken Kanai as Commander. See Triggering Incidents.

Step 2: Join the Incident Channel #

Datadog auto-creates a Slack channel for every incident:

#incident-<incident-id>-<date>
Example: #incident-142-20261005
  • The owning team’s on-call members are added automatically.
  • Ken Kanai is added automatically.
  • All discussion happens in that channel.
  • Pin the Datadog incident link.
  • Invite support for customer-facing incidents.

Step 3: Post Initial Status # 

🚨 INCIDENT DECLARED
Severity: SEV-2
Issue: Extraction jobs not completing
Impact: New uploads stuck in processing
Started: 10:15 UTC
Status: Investigating
Team: Backend
Incident Commander: Ken Kanai
Lead Responder: @your-name
Datadog: https://app.datadoghq.com/incidents/<ID>

Phase 2: Mitigate (Active) #

Goal: Stop customer impact, root cause comes later

Step 1: Investigate #

Use the Incident Response runbook for the technical steps.

Step 2: Post Regular Updates #

Post at the cadence for your severity.

⏱️ STATUS UPDATE (10:30 UTC)
Current status: Investigating
Progress: Queue backlog on extraction consumer
Next steps: Scaling consumers
Next update: 10:45 UTC

Step 3: Keep the Timeline #

  • Add key decisions to the Datadog timeline.
  • Use the Slack message action Add to Datadog Incident for important messages.
  • Record every production change and who made it.

Phase 3: Stable #

Goal: Impact stopped, confirm it stays stopped

Ken moves the incident to Stable in Datadog once:

  • Error rates are back to baseline
  • No new customer reports
  • Mitigation has held for 15 minutes
✅ MITIGATION APPLIED
Action: Rolled back to previous release
Status: Stable, monitoring
Time to mitigate: 18 minutes

Phase 4: Resolved #

Goal: Close the incident and hand over to follow-up

Ken moves the incident to Resolved in Datadog once:

  • The fix is permanent or a tracked follow-up exists
  • All dependent systems are healthy
  • Customer communication is closed
🎉 INCIDENT RESOLVED
Issue: Extraction consumer stuck on poisoned message
Fix: Message moved to DLQ, consumer patched
Duration: 45 minutes
Customer impact: Uploads delayed ~40 minutes
Postmortem: Due within 48 hours

Then archive the Slack channel after the postmortem is linked.

Customer Communication #

The Communications role, assigned by the IC, owns this for customer-facing SEV-1/2.

  1. Acknowledge within 15 minutes
  2. Update every 30-60 minutes
  3. Resolve with a short apology and next steps

Never include customer names or data in shared updates.

Escalation #

Ken is already the IC on every incident. Page @oncall-admin and call Ken directly when:

  • The incident is SEV-1
  • Mitigation has failed for 30 minutes
  • It is a security or data exposure incident
  • Legal or compliance questions arise

See Also #