incident-response

This skill provides structured guidance for handling production incidents.

When to Declare an Incident

An incident should be declared when:

• A production service is degraded or unavailable for users
• Data integrity may be compromised
• A security breach is suspected
• SLA thresholds are at risk of being violated

Roles

| Role | Responsibility | |---|---| | Incident Commander (IC) | Owns the incident lifecycle, coordinates response, makes decisions | | Communications Lead | Keeps stakeholders informed, posts status updates, manages the incident channel | | Technical Lead | Drives root cause investigation, coordinates engineering fixes |

General Triage Steps

1. Acknowledge - Confirm you are responding to the alert
2. Assess - Determine severity using the severity-levels reference
3. Communicate - Open an incident channel and notify stakeholders
4. Escalate - Follow the escalation policy based on severity
5. Mitigate - Apply immediate fixes to restore service
6. Resolve - Confirm the issue is fully resolved and monitoring is clean
7. Postmortem - Schedule and complete a blameless postmortem within 48 hours

Communication Expectations

• Status updates every 15 minutes for SEV1, 30 minutes for SEV2
• All communication happens in the incident Slack channel
• External customer communication is handled by the Communications Lead
• Management is notified per the escalation policy

Available References

• severity-levels.md - Severity definitions (SEV1-SEV4) with impact and response times
• escalation-policy.md - Who to page per severity level and escalation timelines
• postmortem-template.md - Structured template for blameless postmortems

Available Scripts

• page-oncall.sh - Page the on-call engineer via PagerDuty API
• create-incident-channel.sh - Create a dedicated Slack incident channel

Available Assets

• escalation-flowchart.mermaid - Visual flowchart of the escalation process
• severity-ladder.png - The four severity bands, most severe at the top. Fetch this with get_skill_asset when you need to see the ladder rather than read it; on a vision-capable setup it arrives as an image.