Incident Management

Incident Management #

How production incidents are triggered, paged, run and closed using Datadog Incident Management and Datadog On-Call.

Classification: Internal-Only

When to Use These Docs #

  • A Datadog On-Call page reached you
  • You need to declare an incident by hand
  • You are writing or reviewing a postmortem

For the technical debugging steps (kubectl, rollbacks, queues), use the Incident Response runbook alongside these docs.

Ownership #

Four teams own their own monitors and the incidents those monitors raise. The Admin team watches everything.

Team Owns On-Call handle
AI Extraction, LLM, Hinoki, OCR pipelines @oncall-ai
Frontend Webapp, tofu Web, Sales Portal @oncall-frontend
Backend BonsAPI, workers, integrations @oncall-backend
Infra EKS (incl. GPU nodes), RDS PostgreSQL, ElastiCache Redis, Amazon MQ (RabbitMQ), VPC / ALB / WAF / CloudFront, S3, ECR, CI/CD @oncall-infra
Admin All of the above (oversight) @oncall-admin

Incident Commander: Ken Kanai, for every incident. Only Ken can change an incident.

Tooling #

Tool Purpose Access
Datadog On-Call Paging the owning team app.datadoghq.com/on-call
Datadog Incidents Declaring and tracking incidents app.datadoghq.com/incidents
Datadog Monitors Automated triggers app.datadoghq.com/monitors
Slack One channel per incident Auto-created by Datadog
Linear Follow-up action items linear.app

Datadog has replaced PagerDuty. Any remaining PagerDuty reference in these docs or in CI is legacy and is being migrated.

End-to-End Flow # 

P1 monitor fires (owned by AI / Frontend / Backend / Infra)
   ↓
Incident auto-declared in Datadog (Commander: Ken Kanai)
   ↓
Owning team's on-call paged 
   ↓
Slack channel #incident-<incident-id>-<date> created
   ↓
Owning team's on-call members added to the channel
   ↓
Mitigate → Stable → Resolved
   ↓
Postmortem (SEV-1/2: 48 hours, SEV-3: 1 week) → action items in Linear

Documentation Sections #