Incident Response Playbook: A Step-by-Step Guide for SRE Teams
Effective incident response for SRE teams comes down to a repeatable sequence: detect the problem, declare an incident with a clear severity, assign explicit roles, mitigate before you diagnose, communicate on a fixed cadence, resolve and verify, then run a blameless postmortem that produces tracked action items. A good incident playbook writes each of those steps down before the outage happens, so that the people paged at 3 a.m. are executing a plan instead of inventing one.
The rest of this guide walks through each step. We include the artifacts we recommend teams keep in their repositories: severity definitions, role checklists, a status update template, and a postmortem template.
Why Incident Response Breaks Down Without a Playbook
Most incident pain does not come from the technical failure itself. It comes from coordination failures that compound it:
- Nobody is clearly in charge, so three engineers make conflicting changes to the same system.
- Severity is debated mid-incident, which delays escalation.
- Stakeholders ping responders directly for updates, pulling them away from mitigation.
- The root-cause hunt starts too early, while customers are still affected.
- Postmortems produce action items that nobody owns, so the same incident recurs.
Google's Site Reliability Engineering book describes this pattern in its chapter on managing incidents. It contrasts an unmanaged incident with one run under a defined structure built on separation of responsibilities and a recognized command post (Beyer et al., Site Reliability Engineering, O'Reilly, 2016, Ch. 14). A playbook is how you make that structure the default.
Step 1: Define Severity Levels Before You Need Them
Severity drives everything downstream: who gets paged, how often you communicate, and whether a postmortem is mandatory. Define it in terms of customer impact, not internal anxiety.
Keep the definitions in version control next to your alerting configuration:
# incident-severity.yaml
severities:
SEV1:
description: "Critical customer-facing outage or data integrity/security risk"
examples:
- "Checkout or login unavailable for a large share of users"
- "Confirmed data loss or exposure"
page: [primary-oncall, secondary-oncall, incident-commander-rotation]
status_update_interval_minutes: 30
postmortem_required: true
SEV2:
description: "Major degradation or a core feature unavailable for a subset of users"
page: [primary-oncall, incident-commander-rotation]
status_update_interval_minutes: 60
postmortem_required: true
SEV3:
description: "Minor degradation, workaround available, no SLO at risk"
page: [primary-oncall]
status_update_interval_minutes: null
postmortem_required: false
Two rules we recommend writing into the playbook explicitly:
- When in doubt, declare higher. Downgrading a SEV1 to a SEV2 costs very little. Discovering an hour in that an incident was under-declared costs a great deal.
- Tie severity to SLOs where you have them. If an error budget burn rate alert fires, the severity should follow mechanically from the rate, not from a judgment call.
Step 2: Detect and Declare
Detection should come from symptom-based alerts (error rate, latency, saturation against SLOs) rather than from customers. Once an alert fires or someone notices a problem, the on-call engineer's first job is to declare, not to fix.
Declaring means:
- Opening a dedicated channel with a predictable name, for example
#inc-2024-06-12-checkout-latency. - Posting the initial severity and a one-line summary of observed impact.
- Starting a timeline document. Timestamps captured live are far more reliable than reconstructions afterward.
Make declaration cheap. A chat command or CLI that creates the channel, pages the right rotation, and opens the timeline removes friction at the moment people are most stressed:
# Example internal tooling invocation
incident declare \
--sev 2 \
--title "Elevated 5xx on checkout API" \
--service checkout-api \
--page incident-commander
Step 3: Assign Explicit Roles
The most important structural decision in SRE incident management is separating command from hands-on work. The incident roles used in Google's SRE practice and in PagerDuty's publicly available incident response documentation (response.pagerduty.com) share a common core:
- Incident Commander (IC): owns the incident. Makes decisions, delegates, and keeps the team focused. The IC should generally not be typing commands into production.
- Operations Lead: directs the technical investigation and mitigation. For larger incidents, subject matter experts report to this person.
- Communications Lead: owns updates to stakeholders, support teams, and, where applicable, the public status page.
- Scribe: maintains the timeline, records decisions and their rationale, and captures links to dashboards and logs.
On small teams, one person may hold two roles during a SEV3. For SEV1 and SEV2, we recommend insisting on at least an IC who is separate from the person making changes. Handoffs must be explicit: "I am handing IC to Priya, Priya please confirm" in the incident channel, with acknowledgment.
Step 4: Mitigate First, Diagnose Second
The goal during an active incident is to stop customer impact, not to understand it fully. Root cause analysis belongs in the postmortem.
Keep a short list of standard mitigations in the playbook and in each service's runbook:
- Roll back the most recent deploy or configuration change.
- Disable a feature flag associated with the affected path.
- Fail over to a healthy region or replica.
- Shed load or rate-limit non-critical traffic.
- Scale out if the failure is capacity-related and scaling is known to be safe.
The IC should ask one question repeatedly: "What changed, and can we undo it?" Recent changes are a common trigger for incidents, which is why rollback is listed first. Record every mitigation attempt in the timeline, including the ones that did not work. Those failed attempts are often the most informative data in the postmortem.
Step 5: Communicate on a Fixed Cadence
Predictable updates keep stakeholders out of the responders' way. The communications lead posts on the interval defined by severity, even if nothing has changed. "No change, still investigating" is a valid and useful update.
A consistent template removes guesswork:
**[SEV2] Elevated errors on checkout API: Update #3**
**Status:** Mitigating
**Impact:** ~Some users see failed payments on checkout. Browsing unaffected.
**Current action:** Rolling back deploy checkout-api v2.41 to v2.40.
**Next update:** 14:30 UTC or sooner if status changes.
**IC:** @priya | **Comms:** @dan
Keep internal and external communication separate. Internal updates can include technical detail. External updates should describe customer impact and expected next steps in plain language, without speculating on cause.
Step 6: Resolve, Verify, and Hand Off
An incident is resolved when customer impact has ended and metrics confirm it, not when a fix has been deployed. Before closing:
- Confirm the SLIs that triggered the incident have returned to normal and stayed there for an agreed observation window.
- Check for secondary effects: queue backlogs, retry storms, stale caches, or data that needs reconciliation.
- Post a final update with the resolution time and a note that a postmortem will follow.
- If follow-up work is needed overnight, hand it off by name to the next on-call engineer.
Step 7: Run a Blameless Postmortem
The postmortem is where the incident pays for itself. The Site Reliability Engineering book's chapter on postmortem culture emphasizes that a postmortem should focus on contributing causes and systemic fixes, without indicting individuals (Ch. 15). People who fear blame hide information, and hidden information means repeat incidents.
A minimal template:
# Postmortem: <title>
**Date:** | **Severity:** | **Duration:** | **Authors:**
## Summary
Two to three sentences: what happened and the customer impact.
## Impact
Who was affected, how, and for how long. SLO/error budget consumed.
## Timeline (UTC)
Key events from detection through resolution, taken from the live scribe log.
## Contributing Factors
What conditions allowed this to happen? Avoid "human error" as a conclusion;
ask what made the error easy to make.
## What Went Well / What Was Difficult
## Action Items
| Action | Type (prevent/detect/mitigate) | Owner | Due | Ticket |
Three practices separate useful postmortems from paperwork:
- Hold the review within a few business days, while memory is fresh.
- Every action item has a named owner and a ticket. Unowned items are wishes.
- Review open action items regularly in an operations review so they do not silently expire.
Making On-Call Response Sustainable
A playbook only works if the people executing it are rested and trained. Sustainable on-call response includes:
- Reasonable rotation sizes so engineers are not on call too frequently.
- Alert hygiene: every page should be actionable. Delete or downgrade alerts that routinely resolve themselves.
- Runbooks linked directly from alerts, so the first click from a page leads to guidance.
- Game days and tabletop exercises where the team practices the playbook on simulated incidents, including IC handoffs.
- Compensation and recovery time after overnight incidents, set as policy rather than left to goodwill.
If you are building or maturing this function, our overview of managed services and SRE capabilities describes how we approach on-call design, observability, and reliability engineering. Requirements also differ by sector: regulated environments often add notification and evidence obligations, which we cover in our industry-specific guidance.
Rolling Out Your Incident Response Playbook
You do not need to adopt everything at once. A practical sequence:
- Week 1: Agree on severity definitions and commit them to a repository.
- Week 2: Define roles and publish the IC checklist. Train at least two people per team as ICs.
- Week 3: Add the status update template and a declaration workflow, even if it starts as a manual checklist.
- Week 4: Adopt the postmortem template and schedule a recurring action-item review.
- Ongoing: Run a tabletop exercise each quarter and revise the playbook after every significant incident.
For teams that also handle security incidents, NIST's Computer Security Incident Handling Guide (SP 800-61) offers a complementary lifecycle model worth aligning with (NIST CSRC).
The playbook is a living document. Treat every incident as a test of it, and update it when it falls short.
FAQ
What is the difference between an incident playbook and a runbook?
An incident playbook defines the process for any incident: severity, roles, communication, and postmortems. A runbook is service-specific and describes how to diagnose and mitigate particular failure modes, such as failing over a database. You need both, and alerts should link to the relevant runbook.
Should the on-call engineer also be the Incident Commander?
For minor incidents, one person can hold both roles. For high-severity incidents, we recommend separating them so the IC can coordinate, track the big picture, and manage communication while others make changes.
When should we write a postmortem?
At minimum, for every incident at your top two severity levels, and for any incident that consumed significant error budget or involved data integrity. Many teams also write lightweight postmortems for interesting near misses, because they reveal weaknesses before customers feel them.
How do we keep postmortems blameless without losing accountability?
Accountability attaches to action items, not to the person who triggered the incident. Ask what made the failure possible and easy to miss, assign owners to systemic fixes, and track those fixes to completion.
How often should we update the incident response playbook?
Review it after every SEV1 or SEV2 and during each quarterly tabletop exercise. If responders deviated from the playbook during an incident, find out why: either the playbook was wrong or it was unclear, and both are worth fixing.
