Incident Management on BonsAI Documentation /docs/incident-management/ Recent content in Incident Management on BonsAI Documentation Hugo en-us On-Call /docs/incident-management/on-call/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/incident-management/on-call/ <h1 id="on-call"> On-Call <a class="anchor" href="#on-call">#</a> </h1> <p>How Datadog On-Call pages each owning team and who is expected to respond.</p> <p><strong>Classification:</strong> Internal-Only</p> <h2 id="when-to-use-this-page"> When to Use This Page <a class="anchor" href="#when-to-use-this-page">#</a> </h2> <ul> <li>You are joining a team and need paging set up</li> <li>You want to know who gets paged and when</li> <li>You are adding a monitor that should page a team</li> </ul> <h2 id="teams"> Teams <a class="anchor" href="#teams">#</a> </h2> <p>Each team owns its monitors and gets paged for them. The Admin team is notified on every incident.</p> Triggering Incidents /docs/incident-management/triggering/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/incident-management/triggering/ <h1 id="triggering-incidents"> Triggering Incidents <a class="anchor" href="#triggering-incidents">#</a> </h1> <p>Every way an incident can start, and how each source reaches the owning team.</p> <p><strong>Classification:</strong> Internal-Only</p> <h2 id="when-to-use-this-page"> When to Use This Page <a class="anchor" href="#when-to-use-this-page">#</a> </h2> <ul> <li>You want to know why you were paged</li> <li>You are adding a new monitor that should page</li> <li>You spotted a problem and need to raise it</li> </ul> <h2 id="trigger-sources"> Trigger Sources <a class="anchor" href="#trigger-sources">#</a> </h2> <table> <thead> <tr> <th>Source</th> <th>Type</th> <th>Route</th> <th>Result</th> </tr> </thead> <tbody> <tr> <td><strong>P1 Datadog monitor</strong></td> <td>Automated</td> <td><code>@incident-critical</code> + <code>@oncall-&lt;team&gt;</code></td> <td>Incident auto-declared, team paged, Slack channel created</td> </tr> <tr> <td><strong>P2 Datadog monitor</strong></td> <td>Automated</td> <td><code>@oncall-&lt;team&gt;</code></td> <td>Team paged, no incident created</td> </tr> <tr> <td><strong>P3 Datadog monitor</strong></td> <td>Automated</td> <td><code>@oncall-&lt;team&gt;</code></td> <td>Team paged, no incident until declared</td> </tr> <tr> <td><strong>Prod deploy failure</strong></td> <td>Automated</td> <td><code>notify-failures</code> job in <code>prod-deploy.yaml</code></td> <td>Page (PagerDuty today, migrating)</td> </tr> <tr> <td><strong>Sentry alert</strong></td> <td>Automated</td> <td>Sentry → Slack alert channel</td> <td>Engineer declares manually</td> </tr> <tr> <td><strong>Datadog UI</strong></td> <td>Manual</td> <td><strong>Incidents → Declare Incident</strong></td> <td>Incident declared, team paged</td> </tr> </tbody> </table> <h2 id="p1-monitors-end-to-end"> P1 Monitors: End to End <a class="anchor" href="#p1-monitors-end-to-end">#</a> </h2> <p>A P1 monitor <strong>always</strong> triggers an incident. No human decision is needed.</p> Managing Incidents /docs/incident-management/managing-incidents/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/incident-management/managing-incidents/ <h1 id="managing-incidents"> Managing Incidents <a class="anchor" href="#managing-incidents">#</a> </h1> <p>How an incident is run in Datadog from declaration to resolution. Each incident gets its own Slack channel, and Ken Kanai is always the Incident Commander.</p> <p><strong>Classification:</strong> Internal-Only</p> <h2 id="when-to-use-this-page"> When to Use This Page <a class="anchor" href="#when-to-use-this-page">#</a> </h2> <ul> <li>You just declared or acknowledged an incident</li> <li>You are the lead responder for your team</li> <li>You need the status update templates</li> </ul> <h2 id="severity-levels"> Severity Levels <a class="anchor" href="#severity-levels">#</a> </h2> <table> <thead> <tr> <th>Level</th> <th>Definition</th> <th>Response Time</th> <th>Update Cadence</th> <th>Examples</th> </tr> </thead> <tbody> <tr> <td><strong>SEV-1</strong></td> <td>Complete outage or security breach</td> <td>Immediate</td> <td>15 min</td> <td>API down, database unavailable</td> </tr> <tr> <td><strong>SEV-2</strong></td> <td>Major flow impaired</td> <td>15 minutes</td> <td>30 min</td> <td>Login failing, extraction stopped</td> </tr> <tr> <td><strong>SEV-3</strong></td> <td>Partial impact, no workaround</td> <td>1 hour</td> <td>60 min</td> <td>One integration sync failing</td> </tr> </tbody> </table> <p>Only SEV-1 to SEV-3 are used. Minor issues with a workaround become Linear tickets, not incidents.</p> Postmortems /docs/incident-management/postmortems/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/incident-management/postmortems/ <h1 id="postmortems"> Postmortems <a class="anchor" href="#postmortems">#</a> </h1> <p>How every resolved incident turns into learnings and tracked fixes.</p> <p><strong>Classification:</strong> Internal-Only</p> <h2 id="when-to-use-this-page"> When to Use This Page <a class="anchor" href="#when-to-use-this-page">#</a> </h2> <ul> <li>An incident moved to <strong>Resolved</strong></li> <li>You own a postmortem</li> <li>You are running a postmortem review meeting</li> </ul> <h2 id="requirements"> Requirements <a class="anchor" href="#requirements">#</a> </h2> <table> <thead> <tr> <th>Severity</th> <th>Postmortem</th> <th>Deadline</th> <th>Review meeting</th> </tr> </thead> <tbody> <tr> <td><strong>SEV-1</strong></td> <td>Required</td> <td>48 hours</td> <td>Required</td> </tr> <tr> <td><strong>SEV-2</strong></td> <td>Required</td> <td>48 hours</td> <td>Required</td> </tr> <tr> <td><strong>SEV-3</strong></td> <td>Required</td> <td>1 week</td> <td>Optional</td> </tr> </tbody> </table> <p>The Lead Responder from the owning team writes the postmortem. Ken Kanai, as Incident Commander, approves it.</p>