Postmortems

Postmortems #

How every resolved incident turns into learnings and tracked fixes.

Classification: Internal-Only

When to Use This Page #

  • An incident moved to Resolved
  • You own a postmortem
  • You are running a postmortem review meeting

Requirements #

Severity Postmortem Deadline Review meeting
SEV-1 Required 48 hours Required
SEV-2 Required 48 hours Required
SEV-3 Required 1 week Optional

The Lead Responder from the owning team writes the postmortem. Ken Kanai, as Incident Commander, approves it.

Writing the Postmortem #

Step 1: Generate from Datadog #

  1. Open the incident in Datadog → Incidents
  2. Click Generate Postmortem
  3. Pick the team postmortem template
  4. Datadog pre-fills the timeline and metadata

Step 2: Fill the Template # 

## Summary
What happened, in two sentences.

## Impact
Who was affected, for how long, and how badly.
No customer names or PII.

## Timeline (UTC)
10:15 - P1 monitor fired, incident auto-declared SEV-2
10:15 - @oncall-backend paged, #incident-142-20261005 created
10:17 - Page acknowledged by lead responder
10:33 - Mitigation applied, moved to Stable
11:00 - Resolved

## Root Cause
Use 5 Whys. Focus on systems, not people.

## What Went Well

## What Could Be Better

## Action Items
Linked Linear tickets with owners.

Step 3: Create Action Items #

  • Create each action item as a Linear ticket.
  • Label it incident and link the Datadog incident.
  • Every item needs one owner and a due date.
  • Assign the ticket to the owning team.
  • Add the ticket links back to the postmortem.
Category Due
Immediate Done during the incident
Short-term Within 1 week
Long-term Within 1 month

Review Meeting #

Agenda (30-45 min):

  1. Timeline walkthrough (10 min)
  2. What went well (5 min)
  3. What could be better (15 min)
  4. Action items and owners (10 min)

Key principles:

  • Blameless - Systems failed, not people
  • Learning-focused - What would have caught this earlier?
  • Action-oriented - Every gap gets a ticket
  • Shared - Post the final postmortem link in the incident channel

Closing the Loop #

  • Archive the incident Slack channel once the postmortem is linked.
  • Review open incident tickets in weekly planning.
  • Update runbooks and monitors that the incident exposed.

See Also #