Incident Response

Incident Response #

This runbook covers the technical side of production incidents: investigation, mitigation and recovery. For paging, declaring, roles and postmortems, see Incident Management.

When to Use This Runbook #

  • Production service outage or degradation
  • Security incidents or suspected breaches
  • Data integrity issues
  • Customer-impacting bugs
  • System performance degradation
  • Third-party service failures affecting BonsAI
  • Datadog On-Call page received

Incident Severity Levels #

Use the SEV-1 to SEV-3 scale in Managing Incidents. It is the only severity scale.

Incident Response Phases #

Phase 1: Detection & Triage (0-5 minutes) #

Goal: Identify and assess the incident quickly

Step 1: Alert Received #

Incidents may be detected through:

  • Datadog On-Call - Primary paging system (@oncall-<team>)
  • Monitoring alerts (Datadog, CloudWatch)
  • Customer reports (support tickets, Slack)
  • Error tracking (Sentry)
  • Team member observation

Datadog On-Call:

  1. Acknowledge the page within 10 minutes
  2. Declare an incident in Datadog if impact is real
  3. Join the auto-created incident Slack channel

See On-Call and Triggering Incidents.

Step 2: Initial Assessment # 

Quick questions to answer:
1. What is broken? (service, feature, user flow)
2. How many users are affected? (all, some, specific segment)
3. Is this a new issue or recurring?
4. When did it start? (use monitoring dashboards)
5. What changed recently? (deployments, config changes)

Gather initial evidence:

# Check service health
kubectl get pods
kubectl get pods --all-namespaces | grep -v Running

# Check recent deployments
kubectl rollout history deployment/bonsapi-deployment

# Check recent events
kubectl get events --sort-by='.lastTimestamp' | head -20

# Check error rates in Datadog
# Check Sentry for exception spikes

Step 3: Determine Severity #

Use the severity levels to classify the incident.

Severity Decision Tree:

Is production completely down?                → SEV-1
Is a critical user flow broken?               → SEV-2
Is a feature broken with no workaround?       → SEV-3
Is there a workaround or is it cosmetic?      → Linear ticket, no incident

Phase 2: Response & Mitigation (5-30 minutes) #

Goal: Stop the bleeding, restore service

Step 1: Communicate #

Follow Managing Incidents for declaring, the incident channel, status templates and stakeholder updates.

Step 2: Confirm Roles #

Roles are fixed; do not assign an IC. See Managing Incidents.

  • Incident Commander (IC) - Ken Kanai, always
  • Lead Responder - First to acknowledge from the owning team
  • Responder(s) - Owning team on-call, plus teams pulled in
  • Admin team - Oversight and cross-team coordination
  • Communications - Assigned by the IC for customer-facing incidents

Step 3: Investigate #

Use the appropriate runbook:

Symptom Runbook
API errors/downtime Monitoring & Alerting
Pods crashing Kubernetes Debugging
Database issues Database Access
Queue backlog RabbitMQ Management
Deployment issues Deployment Monitoring

Investigation checklist:

# 1. Check service status
kubectl get pods -o wide
kubectl get services
kubectl get ingress

# 2. Check pod logs
kubectl logs -l app=bonsapi --tail=100
kubectl logs -l app=bonsapi --tail=100 | grep -i error

# 3. Check events
kubectl get events --sort-by='.lastTimestamp' | head -30

# 4. Check resource usage
kubectl top nodes
kubectl top pods

# 5. Check external dependencies
# - Database connectivity
# - Redis connectivity
# - RabbitMQ status
# - Third-party APIs (Clerk, Stripe, etc.)

# 6. Review recent changes
git log --oneline --since="2 hours ago"
# Check GitHub Actions for recent deployments

Step 4: Implement Mitigation #

Mitigation strategies (in order of preference):

  1. Quick fix (if root cause is known and simple)

    # Example: Fix config issue
kubectl edit configmap app-config
kubectl rollout restart deployment/bonsapi-deployment

  2. Rollback (if caused by recent deployment)

    kubectl rollout undo deployment/bonsapi-deployment
kubectl rollout status deployment/bonsapi-deployment

  3. Scale up (if resource exhaustion)

    kubectl scale deployment/bonsapi-deployment --replicas=5

  4. Disable feature (if feature causing issues)

    # Use feature flag to disable problematic feature
# Update in Doppler or config

  5. Failover (if infrastructure issue)

    • Route traffic to backup region (if available)
    • Switch to backup database
    • Use cached data

Step 5: Verify Mitigation #

After implementing mitigation:

# 1. Check service health
kubectl get pods
kubectl logs <pod-name> --tail=50

# 2. Test affected functionality
curl https://api.gotofu.com/health
# Test specific endpoints that were broken

# 3. Monitor metrics
# - Error rate should drop to normal
# - Response time should improve
# - Request volume should recover

Update stakeholders:

✅ MITIGATION APPLIED
Action: Rolled back to previous version
Status: Service restored
Impact: API errors stopped, service operational
Time to mitigate: 18 minutes
Next steps: Root cause analysis

Update Datadog:

  1. Add the mitigation to the incident timeline
  2. Move the incident to Stable (not “Resolved” yet)

Phase 3: Resolution & Recovery (30+ minutes) #

Goal: Fully resolve the issue and restore normal operations

Step 1: Confirm Full Recovery #

  • All metrics back to baseline
  • Error rates normal
  • No customer complaints
  • All dependent systems operational

Step 2: Identify Root Cause #

5 Whys Analysis:

Example:
Problem: API returned 502 errors

Why? → BonsAPI pods were not responding
Why? → Pods were out of memory (OOMKilled)
Why? → Memory leak in new code
Why? → Missing memory cleanup in document processing
Why? → Code review didn't catch the leak

Root Cause: Insufficient testing and code review for memory management

Step 3: Implement Permanent Fix #

  1. Create fix branch

    git checkout -b fix/incident-api-oom

  2. Implement fix

    • Fix the root cause
    • Add tests to prevent regression
    • Update documentation

  3. Test thoroughly

    mise run test
mise run ci

  4. Deploy fix

    • Follow standard deployment process
    • Monitor closely during rollout

Step 4: Communicate Resolution # 

🎉 INCIDENT RESOLVED
Issue: Memory leak in document processing
Root Cause: Missing cleanup in new feature
Fix: Deployed patch v2.1.3
Status: Fully resolved
Duration: 45 minutes
Customer impact: 50% of API requests affected
Follow-up: Post-incident review scheduled

Close the Datadog Incident:

  1. Add a final resolution note to the timeline
  2. Move the incident to Resolved
  3. Generate the postmortem (see Postmortems)

Phase 4: Post-Incident Review #

Follow Postmortems. Deadlines, template and approval depend on severity.

Common Incident Scenarios #

Scenario 1: Complete Service Outage #

Symptoms:

  • All API requests failing
  • Health checks failing
  • Pods not responding

Quick Actions:

# 1. Check pod status
kubectl get pods -o wide

# 2. Check recent deployments
kubectl rollout history deployment/bonsapi-deployment

# 3. Rollback immediately
kubectl rollout undo deployment/bonsapi-deployment

# 4. Monitor recovery
kubectl rollout status deployment/bonsapi-deployment

Scenario 2: Database Connection Issues #

Symptoms:

  • Database connection errors in logs
  • Timeouts on database queries
  • 500 errors from API

Quick Actions:

# 1. Check YugabyteDB cluster health in the YugabyteDB console

# 2. Test connectivity from pod
kubectl exec -it <pod-name> -- nc -zv <db-host> <db-port>

# 3. Check connection pool
kubectl logs -l app=bonsapi | grep "connection pool"

# 4. Scale down if overwhelming DB
kubectl scale deployment/bonsapi-deployment --replicas=2

Scenario 3: Third-Party Service Failure #

Symptoms:

  • Errors calling external API (Clerk, Stripe, etc.)
  • Timeouts on authentication/payments
  • Partial functionality broken

Quick Actions:

  1. Verify third-party status

    • Check status page (status.clerk.com, status.stripe.com)
    • Test API directly

  2. Enable graceful degradation

    • Use cached data if available
    • Queue requests for retry
    • Show user-friendly error message

  3. Communicate with users

    • Status page update
    • In-app notification

Scenario 4: Security Incident #

Symptoms:

  • Suspicious activity in logs
  • Unexpected data access
  • Security alert from monitoring

CRITICAL ACTIONS:

  1. Contain the threat IMMEDIATELY

    # Isolate affected systems
kubectl scale deployment/<affected-deployment> --replicas=0

# Rotate potentially compromised credentials
# See Secrets Management runbook

  2. Preserve evidence

    • Capture logs
    • Take snapshots
    • Document findings

  3. Escalate to security team

    • DO NOT discuss in public channels
    • Use secure communication

  4. Follow security incident protocol

    • Assess scope of breach
    • Notify affected parties (if required)
    • Comply with legal/regulatory requirements

Incident Communication #

Internal Communication #

Status updates every 15-30 minutes:

⏱️ STATUS UPDATE (10:30 UTC)
Current status: Still investigating
Progress: Identified high memory usage in API pods
Next steps: Implementing memory limit increase as temporary fix
ETA: 10 minutes

Customer Communication #

For customer-facing incidents:

  1. Initial acknowledgment (< 15 minutes)

    We're aware of an issue affecting API response times.
Our team is investigating and will provide updates soon.

  2. Status updates (every 30-60 minutes)

    Update: We've identified the cause and are implementing a fix.
Impact: ~50% of API requests experiencing delays.
ETA: Service restoration expected within 30 minutes.

  3. Resolution notice

    Resolved: The API issue has been fixed and service is fully restored.
We apologize for the inconvenience and are taking steps to prevent recurrence.

Incident Tools & Resources #

Useful Commands # 

# Quick health check script
kubectl get pods -o wide && \
kubectl get deployments && \
kubectl get services && \
kubectl top nodes && \
kubectl top pods

# Tail logs from all services
kubectl logs -f -l app=bonsapi &
kubectl logs -f -l app=webapp &
kubectl logs -f -l app=bonsai-invoice &

# Watch events
watch kubectl get events --sort-by='.lastTimestamp'

Escalation Path #

When to Escalate #

  • Incident severity is SEV-1
  • Mitigation attempts have failed
  • Issue requires specialized expertise
  • Legal or compliance concerns
  • Security incident

Escalation Contacts #

Follow Managing Incidents.

  1. Owning team (@oncall-<team>)
  2. Another team (@oncall-ai, @oncall-frontend, @oncall-backend, @oncall-infra)
    • Page their handle from the incident channel when you need their expertise
  3. Admin team (@oncall-admin)
    • Cross-team coordination and severity review
  4. Ken Kanai (Incident Commander)
    • Every decision and every change to the incident
    • Security, legal and compliance concerns
    • Assigns Communications for customer-facing updates

Best Practices #

During Incidents #

  • Stay calm - Clear thinking is crucial
  • Communicate clearly - Avoid jargon, be specific
  • Document everything - Actions, findings, decisions
  • Focus on mitigation first - Root cause analysis comes later
  • Don’t guess - Verify before implementing changes
  • Ask for help - Escalate early if stuck

Incident Prevention #

  • Monitor proactively - Don’t wait for customers to report
  • Test thoroughly - Catch issues before production
  • Deploy carefully - Use canary/blue-green deployments
  • Review regularly - Learn from past incidents
  • Practice - Run incident response drills

Post-Incident #

  • Close the loop - Complete all action items
  • Share learnings - Educate the team
  • Update runbooks - Improve documentation
  • Recognize effort - Thank the responders

See Also #