On-Call

On-Call #

How Datadog On-Call pages each owning team and who is expected to respond.

Classification: Internal-Only

When to Use This Page #

  • You are joining a team and need paging set up
  • You want to know who gets paged and when
  • You are adding a monitor that should page a team

Teams #

Each team owns its monitors and gets paged for them. The Admin team is notified on every incident.

Team Datadog team Paging handle Paged for
AI ai @oncall-ai AI-owned monitors
Frontend frontend @oncall-frontend Frontend-owned monitors
Backend backend @oncall-backend Backend-owned monitors
Infra infra @oncall-infra Infra-owned monitors
Admin admin @oncall-admin Every incident, plus escalations

Current Model: Round Robin Paging #

When an alert triggers, Datadog’s round robin picks one member of the owning team and pages them first. The next alert goes to the next member.

Rules:

  • The person who acknowledges is the lead responder.
  • Ken Kanai is always the Incident Commander.
  • If you cannot respond, leave it. Do not acknowledge and walk away.
  • Cross-team help is requested in the incident Slack channel.

Escalation Policy #

Every team’s escalation policy in Datadog On-Call:

Step Notify Escalates if not acknowledged in
1 One team member, picked by round robin 10 minutes
2 The whole owning team 30 minutes
3 Admin team - 
Alert triggers
   ↓
Round robin pages one team member
   ↓ (no ACK in 10 min)
Whole owning team paged
   ↓ (no ACK in 30 min)
Admin team paged

Setting Up Your Paging #

Step 1: Join Your Team #

  1. Open Datadog → Teams and find your team
  2. Ask a team admin (SunTao) to add you if you are not a member
  3. Confirm you appear on the team’s On-Call tab

Step 2: Install the Datadog App #

  1. Download the Datadog app:
  2. Log in with your official tofu email
  3. Allow notifications, including critical alerts so pages bypass Do Not Disturb

Step 3: Configure Notification Preferences #

  1. Open Datadog → Personal Settings → On-Call
  2. Confirm your phone shows under devices after logging into the app
  3. Set the Datadog app as the method for High urgency pages
  4. Set the Datadog app as the method for Low urgency pages
  5. Send yourself a test page from your On-Call profile

Your phone and email should both appear under Channels on your On-Call profile:

On-Call notification channels

Responding to a Page #

Step 1: Acknowledge #

Acknowledge in the Datadog app, Slack or the Datadog UI within 10 minutes. After that the whole team is paged.

Step 2: Join the Incident Channel #

For P1 monitors the incident already exists. You are added to #incident-<incident-id>-<date> automatically.

Step 3: Triage # 

Quick questions to answer:
1. Is this a real customer-facing problem?
2. Which service or flow is affected?
3. Did something just deploy?

Step 4: Decide #

Outcome Action
Real impact Work the incident in its Slack channel
Noise / false positive Tell Ken in the channel, then open a Linear ticket to tune the monitor
Needs another team Page their handle from the channel, for example @oncall-infra

See Also #